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Foreword 


This  effort  was  conducted  under  the  exploratory  development  program  element  0602233N, 
Personnel  and  Training  Technology  (NP2A),  project  number  RM33T23.  The  sponsors  included 
Office  of  Naval  Research  and  AEGIS  Training  Center.  The  objective  of  this  project  was  to  develop 
innovative  and  effective  technologies  to  increase  the  efficiency  and  cost-effectiveness  of  training. 
The  Paperless  Classroom  task  was  initiated  in  FY91  to  explore,  develop,  and  demonstrate  an 
interface  to  large  amounts  of  technical  information  available  in  electronic  form.  A  prototype 
system  was  developed  and  field-tested  under  this  task. 

The  objective  of  this  effort  was  to  evaluate  the  Paperless  Classroom  to  learn  how  use  of  this 
technology  impacts  teaching  and  learning. 


J.  McLACHLAN 
Director,  Classroom  and  Afloat  Training 
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Problem 


Summary 


Information  technology  advances  are  changing  the  way  Navy  personnel  operate  and  maintain 
equipment.  Technical  information  and  documentation  required  to  support  operation  and 
maintenance  of  the  equipment  are  being  put  into  electronic  fonn.  Sailors  are  beginning  to  use 
Interactive  Electronic  Technical  Manuals  (lETM)  onboard  ship.  In  the  classroom,  training 
materials  contain  extensive  cross-references  to  technical  manuals  and  to  other  training  materials  or 
visual  aids.  Instructors  and  students  need  the  capability  to  access  this  electronic  information  during 
classroom  training.  To  address  this  problem,  the  Paperless  Classroom  project  provided  the 
instructor  a  tool  to  facilitate  delivery  of  electronic  training  materials. 

Objective 

The  objective  of  this  effort  was  to  determine  the  effects  on  teaching  and  learning  when 
instructors  lecture  from  electronic  instructional  materials  and  access  visual  aids  through  the  use  of 
computer  interfaces. 

Background 

The  Navy  Personnel  Research  and  Development  Center  (NPRDC)  established  a  memorandum 
of  agreement  with  the  AEGIS  Training  Center  March  1991  to  develop  an  automated  classroom. 
The  NPRDC  exploratory  development  project.  Paperless  Classroom,  was  part  of  the  larger  AEGIS 
Classroom  of  the  Future  effort. 


The  Paperless  Classroom  project  explored  the  use  of  advanced  computer  technology  to  design 
and  develop.  (1)  an  electronic  form  of  the  Instructor  Guide  (IG),  (2)  a  set  of  tools  which  permit  the 
instructors  to  electronically  personalize  the  IG,  (3)  a  presentation  interface  to  the  electronic  IG,  (4) 
an  electronic  technical  publication  interface,  (5)  a  student  workstation/interface;  and  (6)  an 
electronic  form  of  the  Trainee  Guide  (TG). 

Approach 

The  AEGIS  Fundamentals  course  (CDP  034Z)  was  selected  and  developed  for  computer 
display.  This  two  week  course  held  at  AEGIS  Training  Center,  Dahlgren,  VA,  prepares  Fire  Control 
students  for  advanced  training  in  computer  architecture.  Our  objective  was  to  design,  develop,  and 
evaluate  a  system  for  a  small  application  as  a  basis  for  moving  to  larger  applications  in  the  future. 

Phase  I  of  the  Paperless  Classroom  project  designed  and  developed  an  electronic  Instructor 
Guide  (IG).  Phase  H  improved  the  electronic  IG  and  developed  an  electronic  Trainee  Guide  (TG). 

Data  from  observations,  surveys,  interviews,  test  scores,  and  course  critiques  were  collected  to 
obtain  as  much  information  possible  about  the  effect  the  Paperless  Classroom  system  could  have 
on  teaching  and  learning.  Observation  data  covered  the  learning  and  social  environments,  and 
course  activities.  A  survey  was  used  to  gather  demographic  and  opinion  data. 


IX 


Results 


The  Paperless  Classroom  system  was  a  success;  it  was  a  very  useful  tool  for  instructors  to 
access  technical  information  in  electronic  form  as  they  lecture.  The  system  was  reliable  and  easy 
to  use  by  instructors  and  students.  There  was  no  negative  impact  on  learning.  Student  exam  scores 
were  comparable  between  the  Fully  Automated  and  Electronic  IG  classes. 

Conclusions 

This  evaluation  of  the  Paperless  Classroom  was  limited  since  it  was  tested  on  only  one  course. 
However,  this  was  a  good  first  attempt  to  access  the  vast  amount  of  technical  instructional 
materials.  The  instructor  has  been  given  a  new  tool  for  preparing  and  delivering  these  materials. 
An  improved  graphical  interface  has  provided  instructors  and  students  a  better  means  to  review 
schematic  diagrams,  tables,  figures,  and  flow  charts. 

As  the  use  of  electronic  aids  for  classroom  training  increases,  we  expect  improved  instruction 
as  measured  by  increases  in  student  test  scores,  retention  of  course  information,  and  job 
performance.  These  changes  are  expected  because  electronic  aids  to  training  delivery  make  better 
use  of  instructor  and  student  time,  smooth  the  flow  of  information,  and  improve  instructor  control 
over  training  content  and  training  aids.  Electronic  delivery  will  lead  to  reductions  in  publishing 
costs  since  substantial  amounts  of  training  materials  will  no  longer  have  to  be  printed.  Personnel 
requirements,  technical  change  management,  and  curricula  distribution  costs  will  be  reduced. 

Recommendations 

It  is  recommended  that  the  AEGIS  Training  Center  automate  additional  courses  as  more 
interactive  electronic  technical  manuals  (lETM)  become  available.  Consideration  should  be  given 
to  linking  the  Paperless  Classroom  system  to  laboratory  systems.  Instructors  and  students  should 
be  given  more  on-site  training  on  the  use  of  the  Paperless  Classroom  system.  Navy  Instructor 
Training  courses  should  include  delivery  of  electronic  curriculum. 
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Introduction 


Problem 

Information  technology  advances  are  changing  the  way  Navy  personnel  operate  and  maintain 
equipment.  Naval  vessels  are  inundated  with  paper— mostly  technical  information  and 
documentation  required  to  support  operation  and  maintenance  of  the  equipment.  Sailors  are 
beginning  to  use  Interactive  Electronic  Technical  Manuals  (lETMs)  onboard  ship. 

In  training,  the  problem  of  dealing  with  extensive  technical  documentation  is  at  least  as  severe 
as  onboard  ship.  Instructor  and  student  curricular  materials  contain  extensive  cross-references  to 
the  technical  publications  and  to  other  training  materials  or  visual  aids.  The  Paperless  Classroom 
project  addressed  this  problem  by  building  an  interface  to  this  information  and  providing  the 
instructor  a  tool  to  facilitate  delivery  of  electronic  training  materials. 

Objective 

The  overall  objective  of  the  Paperless  Classroom  project  was  to  explore  the  use  of  advanced 
computer  technology  to  design  and  develop:  (1)  an  electronic  form  of  the  Instructor  Guide  (IG), 
(2)  a  set  of  tools  that  permit  the  instructors  to  electronically  personalize  the  IG,  (3)  a  presentation 
interface  to  the  electronic  IG,  (4)  an  electronic  technical  publication  interface,  (5)  a  student 
workstation/interface;  and  (6)  an  electronic  form  of  the  Trainee  Guide  (TG). 

The  objective  of  this  task  was  to  determine  the  effects  on  teaching  and  learning  when 
instructors  lecture  from  electronic  instructional  materials  and  access  visual  aids  through  the  use  of 
computer  interfaces.  The  evaluation  findings  from  Phase  I  reported  in  Morris  (1993),  provided 
information  necessary  to  further  improve  the  system  during  Phase  n. 

Background 

The  Navy  Personnel  Research  and  Development  Center  (NPRDC)  began  working  with  the 
AEGIS  Training  Center  during  FY92  to  develop  an  automated  classroom.  The  NPRDC 
exploratory  development  project.  Paperless  Classroom,  was  part  of  the  larger  AEGIS  Classroom 
of  the  Future  effort.  Classroom  of  the  Future  includes  development  of  Interactive  Electronic 
Technical  Manuals  (lETM)  and  the  Computer  Aided  Presentation  Station  (CAPS)  for  use  at  the 
AEGIS  Training  Center,  Dahlgren,  VA.  The  Paperless  Classroom  project  examined  the  feasibility 
of  building  an  inteiTace  to  the  large  amounts  of  technical  information  available  in  electronic  form 
to  the  instructor  and  students. 

The  AEGIS  Fundamentals  course  (CDP  034Z)  was  chosen  to  receive  the  electronic 
instructional  materials.  Tbis  two  week  course  held  at  AEGIS  Training  Center,  Dahlgren,  VA, 
prepares  students  for  advanced  training  in  computer  architecture.  Tbere  are  over  400  printed  pages 
in  the  IG,  approximately  500  printed  pages  in  the  TG,  and  200  transparencies  of  schematics, 
drawings,  and  tables.  Students  attend  the  AEGIS  Weapons  System  (AWS),  SPY  Radar,  Fire 
Control  System/Operational  Readiness  Test  System  (FCS/ORTS),  Display  or  Computer  courses  as 
follow-on  to  this  course. 
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The  Fundamentals  course  was  authored  using  the  Authoring  Instructional  Materials  (AIM) 
program.  AIM  is  a  Navy  automated  system  for  curriculum  development  and  maintenance.  Course 
materials  are  stored  in  a  relational  database.  This  database  was  a  logical  source  for  retrieving 
course  data  in  a  form  that  could  be  readily  converted  and  used  with  electronic  book  software  to 
produce  a  final  product  for  the  Paperless  Classroom. 

At  AEGIS,  technical  training  is  presented  in  the  IG  and  TG.  The  IG  is  a  detailed  outline  of 
instruction  with  references  to  supporting  technical  publications  and  training  aids,  from  which  the 
instructor  lectures.  It  is  the  instructor's  responsibility  to  “personalize”  the  IG  by  including  detailed 
information  about  the  discussion  points.  The  instructor  may  use  films,  transparencies,  handouts,  or 
technical  manuals  to  illustrate  discussion  points  or  to  enhance  his  lectures.  According  to  AEGIS 
instructors,  they  usually  spend  about  three  weeks  personalizing  the  IG  for  a  two  week  course.  As 
a  newer  version  of  the  IG  became  available,  instructors  copy  the  personalization  into  the  new  IG, 
which  takes  at  least  another  week. 

The  TG  contains  Instruction  Sheets,  which  provide  students  with  more  in-depth  information 
about  the  topics  discussed  by  the  instructor.  The  TG  may  include  several  types  of  Instruction 
Sheets— Information,  Diagram,  and  Problem.  The  Information  Sheet,  much  like  a  book  chapter, 
provides  relevant  information  about  a  topic.  Often  a  Diagram  Sheet  follows  the  Information  Sheet 
to  illustrate  a  topic.  After  students  read  the  Information  Sheet  and  study  the  Diagram,  a  Problem 
Sheet  may  follow  to  further  increase  understanding  of  the  material. 

Traditionally,  the  IG  and  TG  are  paper-based  documents  containing  several  hundred  pages.  For 
even  a  short  course,  there  may  be  large  numbers  of  drawings,  schematics,  and  tables  displayed  as 
transparencies.  In  an  effort  to  save  development  and  printing  costs,  it  was  decided  to  develop 
electronic  instructional  materials  for  use  during  classroom  lecture.  The  hardware  configuration  for 
the  Paperless  Classroom  system  (Appendix  A)  consisted  of  an  Instructor  Station  (Sun  SparcStation 
10  with  two  16"  color  monitors)  and  12  Student  Stations  (Sun  SparcStation  LX  with  one  16"  color 
monitor).  Text  files  were  converted  from  Unix  Trofif  to  Standard  Generalized  Markup  Language 
(SGML)  and  displayed  on  the  computer  via  Dynatext  (Electronic  Book  Technologies,  Inc.) 
software.  Design  and  development  of  the  Paperless  Classroom  system  are  beyond  the  scope  of  this 
report.  See  Morris  &  Dickieson  (1995)  for  more  detailed  information. 

Phase  I  of  the  Paperless  Classroom  project  designed  and  developed  an  electronic  IG.  A 
prototype  electronic  IG  was  delivered  and  evaluated  7-18  December  1992.  Phase  II  improved  the 
electronic  IG  and  developed  an  electronic  TG.  An  evaluation  was  conducted  from  3 1  May  to  10 
June  1994. 


Phase  I— Approach 


As  researchers,  we  realize  it  is  difficult  to  generalize  our  findings  based  on  a  small  sample  of 
students  and  instructors.  It  is  often  very  difficult  to  collect  data  due  to  budget,  time,  and  travel 
constraints.  When  working  with  an  actual  school,  however,  often  there  are  limitations  placed  on 
availability  of  information.  Given  these  constraints,  we  attempted  to  compare  the  various  course 
delivery  methods  as  we  evaluated  the  Paperless  Classroom.  These  course  delivery  methods  are 
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discussed  throughout  this  report  .  We  describe  these  methods  as  “traditional,”  which  refers  to 
paper-based,  and  electronic”  or  “automated,”  which  refers  to  computer-based. 

The  evaluation  process  began  by  first  collecting  baseline  information  from  a  traditional 
Fundamentals  course.  The  evaluator  observed  a  traditional  class  from  17-28  August  1992.  Next, 
the  evaluator  observed  the  same  course  being  taught  with  the  electronic  IG,  10-17  December  1992. 

NPRDC  staff  provided  system  training  for  three  AEGIS  instructors.  This  training  included  an 
introduction  to  the  Unix  operating  system  and  how  to  use  the  electronic  IG  for  personalizing  their 
lessons  and  for  lecturing  in  the  classroom.  NPRDC  developed  a  User's  Manual  to  assist  instructors 
as  they  prepared  their  instructional  materials. 

Subjects 

The  students  were  enlisted  military  personnel,  primarily  from  the  Fire  Control  Technician  (FC) 
rating.  One  student  was  an  Electronics  Technician  (ET).  Military  ratings  are  assigned  for  career 
fields,  just  as  job  titles  are  assigned  for  positions  in  industry.  The  majority  of  the  students  were 
junior  level  (FC3).  The  FCl  and  ET2  are  higher  rating  for  enlisted  personnel.  Table  1  provides 
Student  Rating  for  each  condition,  Traditional  and  Electronic  instruction. 

Table  1 


Phase  I— Student  Rating 


Type  of  Instruction 

FC3 

Number  of  Students 

FC2  FCl 

ET2 

Traditional  (A^  =  20) 

16 

1  2 

1 

Electronic  (N=  19) 

17 

2 

Data  Collection  Methods  and  Instruments 

Many  factors  may  influence  the  success  or  failure  of  new  technology  being  used  in  the 
classroom.  Therefore,  qualitative  and  quantitative  data  from  classroom  observations,  opinion 
surveys,  instructor  interviews,  test  scores,  and  course  critiques  were  collected.  A  description  of  the 
methods  and  instruments  used  to  collect  these  data  follows.  Appendix  B  presents  the  Phase  I 
survey  instruments. 

Observation:  Information  about  the  learning  environment,  classroom  interactions,  and  course 
activities  such  as  the  flow  of  instruction,  time  spent  on  the  various  topics  and  number  of 
transparencies  provided  baseline  data  for  future  evaluations.  This  information  was  gathered 
because  any  of  these  factors  may  impact  teaching  and  learning. 

Survey:  A  survey  was  designed  to  gather  opinions  from  smdents  and  instructors.  There  were 
statements  based  on  a  five-point  scale,  “Strongly  Agree  to  Strongly  Disagree.”  followed  by  open- 
ended  questions.  The  survey  was  administered  to  students  and  instructors  at  the  end  of  the  second 
week  of  instruction.  Another  survey  was  given  to  instructors  to  gather  information  about  the  time 
they  spent  personalizing  die  course. 
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Interviews:  Informal  interviews  were  conducted  with  instructors  to  learn  about  their 
experiences  as  they  personalized  and  delivered  their  training  materials. 

Test  Scores:  Test  scores  were  obtained  for  the  written  quiz  and  final  exam.  The  quiz  counted 
as  10%  of  the  overall  course  grade.  A  score  of  70%  was  required  to  pass  the  final  exam.  These  tests 
were  developed  by  AEGIS  Training  Center  personnel. 

Course  Critiques:  Course  critiques  were  obtained  from  the  students.  The  40-item  critique 
form  was  developed  by  AEGIS  and  covered  the  learning  environment,  training  materials, 
equipment,  instruction,  instructor,  testing,  written  test,  and  performance  test.  The  students  were 
given  a  six-level  ranking  scale,  “Outstanding”  to  “Not  Applicable,”  from  which  to  select  their 
responses. 


Phase  I— Results 


Traditional  Instruction 

Observation 

Learning  Environment:  Training  was  held  in  a  long  narrow  classroom  with  tables  and  chairs 
to  accommodate  20  students.  See  Figure  1  for  a  layout  of  the  classroom.  The  room  was  in  an 
environmentally  controlled  building;  comfortably  air  conditioned  and  heated.  Security  was  of 
utmost  importance  throughout  the  building.  All  of  the  doorways  were  electronically  controlled  and 
the  classroom  door  was  equipped  with  a  combination  lock.  There  were  no  windows.  The  room  was 
well  insulated.  The  walls  were  off-white  in  color  and  the  carpet  was  beige.  Large  white  boards  were 
mounted  on  the  front  wall  (behind  the  instructor)  and  at  the  back  of  the  room.  These  walls  were 
painted  deep  orange.  The  fluorescent  lighting  was  adequate  and  pleasant. 

An  overhead  projector  and  a  pull  down  film  screen  were  in  the  front  left  of  the  room.  Two  TV 
monitors  were  ceiling  mounted  in  the  upper  right  comer  and  right  side  midway  of  the  room.  A  VCR 
and  two  closed-circuit  cameras  were  placed  on  the  left  side  of  the  room.  Bookcases  were  on  the 
left  of  the  room  adjacent  to  student  tables.  Technical  manuals  and  course  materials  were  stored  in 
these  bookcases. 

The  chairs  were  uncomfortable  straight  gray  armchairs  with  stationary  legs.  Each  student  had 
a  table  with  adequate  space  for  course  materials.  The  instmctor  taught  from  a  podium  and  table 
located  at  the  front  of  the  room.  The  instmctor  preferred  to  walk  down  the  aisle  between  the  groups 
of  tables  to  have  closer  contact  with  the  students  and  to  make  sure  they  could  hear  the  lecture.  He 
often  carried  the  IG,  TG,  or  textbook  as  he  discussed  the  material. 

Classroom  Interaction:  There  was  respect  for  military  rank  between  the  instmctor, 
indoctrination  speakers,  and  students.  The  instmctor  was  an  enlisted  member  of  the  military.  He 
was  very  professional  and  confident  with  the  material.  Students  commented  that  he  was  very 
helpful  when  responding  to  their  questions.  The  instmctor  interacted  very  well  with  the  students. 
The  military  and  civilian  personnel  who  spoke  to  the  class  during  the  course  indoctrination  were 
genuinely  concerned  about  the  morale  and  well  being  of  the  students  and  their  families;  offering 
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counseling  and  assistance  as  needed.  By  the  second  day,  students  began  to  feel  comfortable  and 
talked  freely  among  themselves  and  with  the  instructor.  One  of  the  students  was  the  class  leader 
and  had  responsibility  for  the  duty  roster.  Some  of  the  students  had  served  in  the  fleet  and  shared 
their  experiences  with  other  students. 


Camera 


Overhead 

Projector 


White  Board 


Instructor  Podium 


TV 


Figure  1.  Phase  I-- Traditional  instruction  classroom  layout. 
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Course  Activities:  The  evaluator  was  present  during  the  entire  two  week  course.  The  course 
was  conducted  during  normal  duty  hours  the  first  week.  During  the  second  week,  the  course  was 
conducted  from  noon  into  the  evening.  The  first  three  days  included  indoctrination  to  the  base  and 
the  training  center,  welcoming  by  various  base  organizations,  a  discussion  about  safety,  and  time 
to  take  care  of  housing,  dental,  or  medical  needs.  Lecture  began  on  the  fourth  day. 

Topics  covered  during  the  course  were  Digital  Systems,  CPU  and  Microprogrammable  Control 
Units,  Computer  Interfaces,  Memory  Architecture,  Executive  Operating  Systems,  Utility 
Programs,  Diagnostic  Programs,  Peripheral  Devices,  and  General  Physical  Description  of  AEGIS 
Combat  System.  There  were  20  transparencies  shown  from  an  overhead  projector,  some  more  than 
once,  to  illustrate  topics  during  lecture.  A  30-minute  video  about  safety  was  shown.  The  white 
board  was  used  extensively  to  further  explain  a  discussion  point  during  the  lectures.  Seven  wall 
charts  were  used  to  discuss  the  general  layout  of  the  various  ships,  overall  system  architecture,  and 
types  of  personnel  onboard  each  ship.  Students  completed  a  laboratory  exercise  on  the  AEGIS 
Combat  System  Interactive  Trainer  (ACSIT).  At  the  end  of  the  course,  AEGIS  course  critiques 
were  completed  by  students. 

One  written  quiz,  consisting  of  five  multiple  choice  questions,  was  given  during  the  second 
week  of  instruction.  Students  could  use  their  books  and  notes.  A  written  exam,  which  was  open- 
book,  was  given  the  last  day  of  the  course. 

Survey:  Since  this  was  a  paper-based  course,  the  opinion  survey  on  the  use  of  our  system  was 
not  applicable. 

Interviews:  Informal  interviews  were  conducted  with  two  instructors.  The  instructors  were 
very  confident  with  the  material  and  enjoyed  teaching.  Instructors  reported  it  took  them  as  long  as 
three  weeks  to  personalize  the  paper-based  IG.  This  was  a  substantial  amount  of  time  given  the 
course  is  only  two  weeks  long.  When  the  course  is  revised,  their  notes  have  to  be  transferred  to  a 
new  copy  of  the  paper-based  IG.  The  personalization  literally  had  to  be  re-written  on  the  newly 
printed  IG. 

Test  Scores:  For  this  traditional  group  {N  =  20),  student  final  exam  scores  ranged  from  75- 
100%,  class  mean  =  88.75.  Quiz  scores  ranged  from  60-100%,  class  mean  =  94.  The  cumulative 
course  mean  =  89.28.  This  mean  included  the  quiz  scores. 

Course  Critiques:  The  majority  of  students  found  the  classroom  environment  to  be  excellent; 
the  quality  of  training  materials  good  to  excellent;  and  equipment,  instruction,  ability  of  instructor, 
testing  policy,  and  clarity/adequacy  of  the  written  test  to  be  excellent.  Students  were  concerned 
about  the  organization  of  the  subject  matter.  The  order  of  topics  taught  did  not  agree  with  that  of 
the  textbook,  resulting  in  a  lot  of  “jumping  around”  in  the  text. 

Electronic  Instruction 

Observation 

Learning  Environment:  Training  was  conducted  in  a  rectangular,  somewhat  crowded, 
classroom  with  tables  and  chairs  to  accommodate  20  students.  See  Figure  2  for  the  classroom 
layout.  The  building  was  comfortably  air  conditioned  and  heated.  Security  was  of  utmost 
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importance  throughout  the  building.  All  of  the  doorways  were  electronically  controlled  and  the 
classroom  door  was  equipped  with  a  combination  lock.  There  were  no  windows.  The  instructor 
taught  from  a  podium,  which  could  be  raised,  lowered,  or  tilted  at  the  front  of  the  classroom.  The 
keyboard  and  mouse  were  placed  on  this  podium.  The  computer  monitor  was  located  on  a  table  to 
the  right  of  the  raised  podium.  The  room  was  well  insulated.  The  walls  and  carpet  were  grey.  The 
chairs  were  armless  with  chrome  trim,  stationery  legs,  and  blue  cushions.  Large  white  boards  were 
mounted  on  the  front  wall  behind  the  instructor  podium.  The  white  board  was  used  extensively  to 
illustrate  a  topic.  The  fluorescent  lighting  was  adequate  and  pleasant.  An  overhead  projector,  a  pull 
down  film  screen,  and  two  TV  monitors  (mounted  in  the  upper  right  comer  and  right  side  midway 
of  the  room)  were  in  the  front  of  the  classroom.  A  VCR  and  two  closed-circuit  cameras  were  placed 
at  the  back  of  the  room.  The  course  was  video-taped  to  allow  the  system  developers  to  review 
sessions  as  they  made  improvements.  Bookcases  were  on  the  right  and  left  of  the  room  adjacent  to 
student  tables.  Technical  manuals  and  course  materials  were  stored  in  these  bookcases. 

Each  student  had  a  table  with  adequate  space  for  course  materials  to  be  stowed  in  a  tray 
underneath  the  table  top.  The  tables  were  arranged  with  an  aisle  down  the  left  side.  The  evaluator 
sat  at  a  table  in  the  back  right  of  the  class.  Another  instructor  sat  at  a  table  to  the  evaluator's  left. 
He  was  auditing  the  course  to  prepare  for  teaching  the  following  month.  There  were  two  computers 
at  the  back  of  the  room.  These  were  instmctor  workstations  installed  to  give  instructors  an 
opportunity  to  personalize  their  lectures  and  to  become  familiar  with  using  the  electronic  IG 
program. 

Classroom  Interaction:  The  instmctor  was  an  enlisted  member  of  the  military.  He  had  four 
years  of  computer  experience  and  had  been  teaching  for  three  years.  He  had  a  good  rapport  with 
the  students  and  was  confident  using  the  electronic  IG  as  he  lectured.  The  instmctor  interacted  well 
with  students  and  responded  to  their  questions  with  ease. 

After  a  couple  of  days,  students  interacted  freely.  The  class  leader  was  responsible  for  the  duty 
roster  and  kept  students  informed  of  upcoming  events  on  a  daily  basis.  Some  of  the  students  had 
fleet  experience.  Because  of  this  experience,  other  students  felt  comfortable  asking  them  questions 
relevant  to  equipment  and  duty  onboard  the  ships.  Most  of  the  students  had  just  attended  their  first 
level  of  training  at  “A”  school  and  did  not  have  fleet  experience. 

Course  Activities:  The  evaluator  was  present  during  the  entire  two  week  course.  The  course 
schedule  was  the  same  as  that  for  the  traditional  course.  During  this  course,  an  electronic  IG,  a 
paper-based  TG,  technical  manual,  and  textbook  were  used. 

There  were  no  delays  during  the  course,  no  hardware  or  software  failures.  The  instmctor  was 
very  comfortable  with  the  electronic  IG  and  seemed  to  enjoy  using  it.  Noticing  something  he 
wanted  to  add  to  a  discussion  point,  he  personalized  his  lesson  “on  the  spot,”  without  intermpting 
the  class. 

The  same  topics  as  those  for  the  traditional  course  were  taught.  The  only  difference  noted  was 
that  fewer  transparencies  were  shown  to  illustrate  topics  during  lecture.  As  in  the  traditional 
course,  the  white  board  was  used  extensively  to  further  explain  a  topic.  Five  wall  charts  were  used 
to  discuss  the  general  layout  of  the  various  ships  and  overall  system  architecture  onboard  each  type 
of  ship.  A  30-minute  video  about  safety  was  shown  during  the  course. 
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Figure  2.  Phase  I— Electronic  instruction  classroom  layout. 


The  same  procedure  for  testing  was  followed  for  this  group  as  for  the  traditional  instruction 
group.  Students  completed  a  laboratory  exercise  on  the  AEGIS  ACSIT.  At  the  end  of  the  course, 
AEGIS  course  critiques  were  completed  by  students. 

Survey 

Instructor:  To  summarize  survey  findings  (Appendix  B,  Table  B-1),  the  instructor  supported 
the  use  of  electronic  aids  for  personalizing  his  lecture  and  for  delivering  classroom  training.  He  had 
used  computers  for  about  four  years  and  had  three  years  teaching  experience.  The  instructor 
strongly  agreed  that  new  media  should  be  used  in  the  classroom,  that  he  could  save  time  preparing 
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the  IG,  and  that  use  of  the  system  would  be  useful  for  the  life-cycle  management  of  the  course.  He 
expressed  confidence  with  using  the  system  as  a  tool  for  course  development  and  lecture. 

Student:  A  summary  of  student  (N  =  19)  survey  responses  (Appendix  B,  Table  B-2)  revealed 
that  more  than  half  of  the  smdents  had  computer  experience,  mostly  four  years  or  less,  but  few  had 
used  a  computer  in  a  classroom  setting.  Six  of  the  students  said  they  had  no  computer  experience, 
and  one  had  experience  of  less  than  a  year.  The  majority  of  students  favored  the  use  of  media  in 
the  classroom  and  did  not  think  the  flow  of  instruction  was  adversely  affected  by  the  instructor 
teaching  from  an  electronic  IG.  Some  students  selected  an  “undecided”  response  to  the  statements 
about  hardware  and  software  problems  because  they  could  not  view  the  Electronic  IG.  They  based 
their  responses  on  the  instructor's  reaction  to  any  problems  related  to  the  system. 

Suggestions  made  by  five  students  for  improving  the  use  of  this  technology  in  the  classroom 
included  (1)  eliminating  transparencies  by  displaying  the  graphics  on  a  computer,  (2)  installing 
computers  for  students,  and  (3)  including  a  study  module  on  student  computers.  Other  comments 
made  by  students  were  “the  paperless  classroom  was  a  great  idea  whose  time  had  finally  come” 
and  that  automating  the  class  would  make  learning  more  interesting  because  of  the  interaction. 
They  also  thought  it  was  difficult  to  follow  where  the  instructor  was  in  the  material  because  the 
textbook  topics  were  not  in  the  same  sequence  as  the  electronic  IG.  Students  wanted  to  see 
networked  student  workstations  in  their  quarters  so  they  could  use  the  system  for  extra  study  and 
homework. 

Interviews:  The  instmctor  suggested  the  electronic  IG  be  improved  by  organizing  it  by  topics 
so  the  scroll  bar  reflected  where  he  was  in  the  topic  rather  than  in  the  entire  IG.  This  would  make 
it  easier  to  judge  how  much  of  the  topic  is  remaining  and  how  much  time  is  required  to  finish  the 
discussion.  He  suggested  a  status  line  be  added  at  the  top  to  inform  the  user  of  the  location  within 
a  topic.  Another  suggestion  was  that  the  bookmark  icon  be  a  different  color  to  distinguish  it  from 
other  icons  used  for  personalization.  A  bookmark  is  a  place  holder  noted  by  an  icon  displayed  next 
to  associated  text  on  the  computer  screen.  The  instructor  thought  the  Table  of  Contents  (TOC) 
should  close  automatically  instead  of  requiring  the  instructor  to  close  it.  He  identified  a  problem 
when  using  the  arrow  up  or  down  keys.  As  the  computer  screen  refreshed,  text  was  displayed 
improperly.  If  he  used  the  Page  Down  key  to  refresh,  the  problem  was  gone.  All  of  these 
suggestions  and  problems  noted  were  given  to  the  system  developers  for  consideration  as  the 
program  evolved. 

Test  Scores:  For  the  electronic  instruction  class  (N  =  19),  student  final  exam  scores  ranged 
from  75-100%,  class  mean  =  89.21.  Quiz  scores  ranged  from  53-100%,  class  mean  =  80.  The 
cumulative  course  mean  =  88.29.  This  mean  included  quiz  scores. 
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A  comparison  of  test  scores  for  the  Traditional  (N  =  20)  and  Electronic  (N  =  19)  instruction 
classes  is  shown  in  Table  2.  The  results  of  an  analysis  of  variance  (ANOVA)  found  no  significant 
differences  between  these  exam  scores,  F(l,37)  =  .0401, p  >  .05. 

Table  2 

Phase  I— Student  Scores 


Class  Average  (%) 

Cum. 

Type  of  Instruction 

Quiz 

Exam 

Course 

Traditional  (^=20) 

94 

88.75 

89.28 

Electronic  (^=  19) 

80 

89.21 

88.29 

Course  Critiques:  Students  found  the  classroom  environment  to  be  excellent;  the  quality  of 
training  materials  good  to  excellent;  and  equipment,  instruction,  ability  of  instructor,  testing  policy, 
and  clarity/adequacy  of  the  written  test  good  to  outstanding.  Students  commented  negatively  about 
the  logical  arrangement  of  the  subject  matter.  The  IG  and  TG  did  not  follow  the  same  sequence  of 
topics  as  that  of  the  textbook  used  in  class.  Some  students  commented  that  the  written  test  was  of 
poor  printed  quality.  The  course  managers  seriously  considered  comments  and  recommendations 
made  by  students. 


Phase  n-- Approach 


The  Paperless  Classroom  system  was  ready  for  field  testing  May  1994.  Because  of  an  increase 
in  the  number  of  students  attending  the  Fundamentals  course  between  January  and  June  1994, 
training  was  conducted  in  two  classrooms,  one  equipped  with  an  Electronic  IG  (no  student 
workstations)  and  one  Fully  Automated  (instructor  and  student  workstations).  The  term  "fully 
automated"  means  the  IG  was  displayed  on  the  instructor  workstation,  the  TG  on  the  student 
workstation,  and  graphics  on  both  workstations. 

A  chief  complaint  during  prior  traditional  classes  was  that  the  course  materials  (IG,  TG,  and 
textbook)  did  not  cover  topics  in  the  same  sequence.  Students  frequently  commented  about  the 
confusion  created  by  "jumping  around"  through  the  course  materials.  The  course  was  not  rewritten, 
but  the  text  was  incorporated  into  the  TG  and  the  sequence  of  topics  put  in  better  order.  All  of  the 
transparencies,  tables,  and  figures  were  re-authored  and  included  in  the  computer  system.  Thus,  the 
paperless  classroom  became  a  reality. 

This  evaluation  was  conducted  by  collecting  information  from  one  Electronic  IG  and  one  Fully 
Automated  class.  One  evaluator  observed  the  Electronic  IG  class  conducted  from  23  May  to  3  June 
1994.  Another  evaluator  observed  the  Fully  Automated  class  conducted  from  31  May  to  10  June 
1994.  Their  goals  were  to  observe  the  course  being  taught  and  to  collect  data  relevant  to  use  of  the 
system,  learning  environment,  student/instructor  interaction,  and  opinions.  A  survey  was 
administered  to  collect  demographic  and  opinion  data. 
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NPRDC  staff  provided  system  training  for  AEGIS  instructors.  This  training  included  how  to 
use  the  Electronic  IG  for  personalizing  their  lessons  and  for  lecturing  in  the  classroom.  The  system 
User's  Manual,  delivered  during  Phase  I,  was  revised  and  provided  to  the  instructors.  This  manual 
was  a  reference  to  assist  instructors  as  they  prepared  their  instructional  materials  using  the 
Paperless  Classroom  system. 

Subjects 

The  students  were  enlisted  military  personnel  of  the  Fire  Control  Technician  (FC)  rating.  Table 
3  shows  the  majority  of  the  students  were  junior  level  (FCSN  and  FC3).  The  FCl  is  higher  rating 
among  these  enlisted  personnel. 


Table  3 


Phase  n— Student  Rating 


Number  of  Students 

Type  of  Instruction 

FCSN 

FC3 

FC2 

FCl 

Electronic  IG(A^=  22) 

5 

12 

4 

1 

Fully  Automated  (N=  9) 

3 

3 

2 

1 

Data  Collection  Methods  and  Instruments 

Data  based  on  classroom  observations,  survey  of  students  and  instructors,  interviews  with 
instructors,  and  test  scores  were  collected.  The  methods  and  instruments  used  to  collect  these  data 
are  described  below.  Appendix  D  present  the  survey  instruments. 

Observation:  Evaluators  were  present  to  observe  the  Electronic  IG  course  (paper-based  TG, 
no  student  workstations)  and  the  Fully  Automated  class  (electronic  IG/TG  and  student 
workstations).  The  flow  of  instruction,  time  spent  on  the  various  topics,  and  interaction  between 
the  instructor  and  students  provided  valuable  information  about  use  of  the  Paperless  Classroom 
system. 


Survey:  The  survey  administered  during  Phase  I  was  revised  to  include  open-ended  questions 
asking  students  and  instructors  to  list  what  they  liked  most  and  least  about  the  classroom 
interaction,  instructional  strategies,  equipment,  subject  matter,  instructor,  and  facilities.  The  survey 
was  administered  to  students  and  instructors  at  the  end  of  the  course.  Instructors  were  also  given  a 
survey  regarding  the  time  spent  personalizing  the  paper-based  and  electronic  IGs. 

Test  Scores:  Test  scores  were  obtained  for  students  attending  the  classes  we  observed  during 
the  May-June  1994  time  frame.  In  addition,  test  scores  were  obtained  for  recent  paper-based,  fully 
automated,  and  electronic  IG  classes  that  could  be  relevant  to  this  evaluation  (see  Appendix  C, 
Table  C-1). 

Course  Critiques:  AEGIS  course  critiques  were  not  available  during  Phase  II  evaluation. 
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Phase  II— Results 


Data  from  the  observations,  surveys,  and  interviews  are  reported  for  the  Electronic  IG  class  and 
the  Fully  Automated  class. 

Electronic  IG  Instruction 

Observation 

Learning  Environment:  The  Electronic  IG  instruction  was  held  in  the  same  location  as  the 
Phase  I  Traditional  Instruction  class  previously  observed.  The  classroom  layout,  as  shown  in 
Figure  1,  was  also  the  same.  Students  were  given  two  binders  containing  the  TG  and  technical 
manual,  and  the  textbook  (Mano,  1993).  These  materials  were  used  by  students  during  lecture  and 
could  be  taken  home  for  study.  There  were  no  student  workstations  in  this  classroom. 

The  instructor  lectured  from  the  electronic  IG.  The  computer  was  placed  on  a  table  to  the  left 
of  a  podium,  at  the  front  of  the  room.  The  paper-based  TG  and  transparencies  of  the  graphics  were 
used  during  the  lecture.  A  cable  was  accidentally  disconnected  during  a  lecture,  but  no  other 
equipment  or  problem  failures  were  noted. 

Classroom  Interaction:  The  instructor  was  an  enlisted  member  of  the  military.  He  was  very 
professional,  had  been  teaching  for  a  year,  but  had  never  lectured  from  a  computer.  Students 
commented  that  the  instructor  was  very  helpful  when  they  asked  questions.  The  instructor 
interacted  very  well  with  the  students.  This  was  the  first  Fundamentals  course  to  have  a  female 
student  in  attendance.  The  male  students  accepted  her  and  everyone  felt  comfortable.  By  the 
second  day,  students  began  to  talk  freely  among  themselves  and  with  the  instructor.  One  of  the 
students  was  the  class  leader  and  had  responsibility  for  the  duty  roster.  Quite  a  few  students  had 
served  in  the  fleet  prior  to  this  training.  They  shared  their  experiences  with  other  students. 

Course  Activities:  The  evaluator  was  present  during  all  of  the  instructional  sessions.  The 
course  was  conducted  during  normal  duty  hours  the  first  week.  During  the  second  week,  the  course 
was  conducted  from  noon  into  the  evening.  The  first  three  days  included  indoctrination  to  the  base 
and  the  training  center,  a  topic  about  safety,  and  time  to  take  care  of  personal  business.  Lecture 
began  on  the  fourth  day.  The  same  topics  were  taught  as  those  in  the  Fully  Automated  class.  As  in 
the  other  Fundamentals  classes,  a  30-minute  safety  video  was  shown.  The  white  board  was  used 
extensively  to  further  explain  a  discussion  point  during  the  lectures. 

The  same  testing  procedures  as  for  the  other  Fundamentals  classes  were  followed;  one  written 
quiz  and  a  final  exam. 

Survey 

Instructor:  To  summarize  results  of  the  survey  (Appendix  C,  Table  C-2),  the  instructor  had 
about  nine  years  computer  experience,  but  had  no  previous  experience  using  a  computer  in  a 
classroom  setting.  He  had  one  year  teaching  experience  and  was  supportive  of  having  new  media 
in  the  classroom;  he  felt  confident  using  the  computer.  In  spite  of  this  confidence,  he  disagreed  that 
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he  could  personalize  the  IG  with  ease  using  the  system  or  that  it  would  save  development  time. 
And,  he  was  undecided  about  using  computers  in  a  classroom  to  enhance  training. 

When  asked  “Do  you  have  suggestions  for  improving  the  use  of  this  technology  in  the 
classroom?,”  the  instructor  suggested  he  would  like  the  capability  to  search  for  keywords  in  the 
course  personalization  and  to  have  more  colors  available  for  highlighting  personalization.  The 
present  system  does  not  support  searching  for  keywords  within  the  personalization  notes,  only  in 
the  basic  IG  outline.  The  personalization  text  can  be  shown  in  various  colors  (the  system  now 
presents  this  text  in  blue),  but  a  change  could  be  made  by  the  system  designer.  The  instructor  would 
also  like  to  have  the  on-line  TOC  broken  down  to  discussion  point  level.  Presently  the  TOC 
displays  the  IG  in  part,  section,  and  topic  levels.  An  optional  TOC  can  be  retrieved  that  lists  figures 
and  tables  within  these  levels.  The  instructor  also  stated  he  would  like  to  have  the  capability  to  add 
his  own  graphics  and  to  transfer  graphics  from  the  computer  to  an  overhead  screen. 

To  gain  an  overall  opinion  about  the  learning  environment,  the  instructor  was  given  various 
categories  (classroom  interaction,  equipment  used,  subject  matter,  facilities)  and  asked  to  comment 
about  what  he  liked  “most”  and  “least”  about  the  course.  He  stated  the  new  system  “forced  him  to 
really  study  curriculum.  He  reported  that  the  system  didn't  crash  in  class  and  that  other  than  the 
use  of  a  computer  for  lecture,  there  were  no  differences  between  this  class  and  a  paper-based  class 
as  far  as  facilities  and  subject  matter.  At  times,  the  instructor  found  it  difficult  to  respond  to  student 
questions  because  he  could  not  find  the  answer  in  the  Electronic  IG  promptly.  The  instructor  felt 
confined  to  the  computer.  He  was  used  to  walking  around  the  room  when  teaching  from  a  paper- 
based  IG.  The  instructor  thought  it  took  too  much  time  each  day  to  prepare  for  lecture.  He  had  to 
log  into  the  computer,  access  the  program,  and  find  his  place  in  the  IG.  He  also  said  he  would  like 
to  see  a  different  podium/workstation  layout.  The  instructor  stated  he  would  like  his 
personalization  notes  to  be  a  preset  size  so  he  would  not  have  to  resize  the  window  each  time.  The 
system  designers  felt  the  personalization  text  would  not  be  very  lengthy.  A  small  window  is  first 
displayed  for  input  of  personalization  text,  but  the  present  system  design  gives  the  instructor  the 
capability  to  size  the  window. 

Student:  Twenty-two  students  responded  to  the  survey.  See  Appendix  C,  Table  C-3  for  the 
survey  results.  The  majority  of  students  (17)  had  computer  experience,  but  most  had  not  used  a 
computer  in  a  classroom  setting.  Students  had  used  computers  for  quite  a  few  years:  with  close  to 
half  having  6-12  years  experience  and  only  five  had  no  computer  experience.  Students  were 
supportive  of  using  new  media  in  the  classroom  and  thought  that  the  instructor  had  few  problems 
as  he  lectured  from  the  electronic  IG.  They  felt  the  system  was  reliable  and  thought  use  of  the 
electronic  IG  did  not  adversely  affect  their  learning. 

To  gain  an  overall  opinion  about  the  learning  environment,  the  students  were  given  various 
categories  and  asked  to  comment  about  what  they  liked  “most”  and  “least”  about  the  course. 

When  asked  “What  do  you  like  MOST  about  the  course?”  with  respect  to  these  categories, 
students  made  the  following  comments. 

Classroom  Interaction:  Six  students  felt  there  was  a  lot  of  classroom  interaction/discussion 
and  that  they  were  acknowledged  when  they  had  questions.  One  student  thought  other  students 
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asking  questions  helped  him  understand  material.  One  student  commented  that  he  liked  that 
personal  experiences  of  students  were  allowed  during  discussions.  Two  students  thought  the  course 
flowed  smoothly. 

Instructional  Strategies:  Four  students  said  the  course  was  “excellent,”  “successful,”  “moved 
very  quickly,”  and  with  “good  structure.” 

Equipment  Used:  Three  students  stated  that  the  computer  seemed  to  help  instructor  discuss 
the  material. 

Subject  Matter/Curriculum:  Five  students  commented  that  the  course  was  “Very  interesting 
to  leam.”  One  said  the  course  was  a  good  introduction  to  computer  architecture. 

Instructor:  Nineteen  students  thought  the  instmctor  explained  the  material  well,  seemed  quite 
confident,  and  well  versed  in  the  subject  matter.  They  also  thought  he  clarified  and  answered  dl  of 
their  questions  completely.  One  student  said  the  instructor  was  motivated  and  energetic. 

Facilities:  Twelve  students  commented  that  the  facilities  were  excellent  and  comfortable. 

When  asked  “What  do  you  like  LEAST  about  the  course?”  given  the  following  categories, 
there  were  few  comments. 

Classroom  Interaction:  One  student  commented  there  needed  to  be  more  diagram  tracing.  To 
clarify,  he  meant  the  instructor  “traces”  by  hand,  with  a  pen,  on  a  schematic  drawing  as  he  discusses 
a  topic.  This  is  commonly  done  on  transparencies  when  displaying  with  an  overhead  projector.  For 
instance,  the  instructor  may  trace  the  flow  of  elecUicity  through  a  circuit.  One  student  stated  there 
was  not  enough  interaction  due  to  time  constraints. 

Instructional  Strategies:  Three  students  commented  on  this  category.  One  said  he  would  like 
more  time  to  cover  the  material,  one  said  there  should  be  more  student  participation,  and  the  other 
student  stated  there  were  “Not  enough  visual  aids  to  add  interest  in  times  of  dry  material.” 

Equipment  Used:  Six  students  commented  about  equipment.  Three  thought  student 
workstations  would  greatly  improve  classroom  participation.  During  this  instruction,  only  the 
instructor  had  a  workstation.  One  commented  that  the  transparencies  were  not  very  state-of-the- 
art,  and  he  suggested  the  use  of  computer-based  graphics. 

Subject  Matter/Curriculum:  Seven  students  commented  that  the  subject  matter  was  “dry, 
very  repetitive,  not  presented  in  a  way  that  it  can  be  discussed  or  practiced.” 

Instructor:  No  comments. 

When  asked  “Do  you  have  suggestions  for  improving  the  use  of  this  technology  in  the 
classroom?”  four  students  offered  suggestions.  Two  students  suggested  providing  computers  to 
each  student  and  adding  lots  of  visuals.  One  student  suggested  better  placement  of  the  instructor's 
computer  monitor.  He  thought  the  addition  of  a  large  screen  monitor  in  the  back  of  the  room  would 
allow  the  instructor  more  flexibility  as  he  walked  through  the  classroom  lecturing.  Another  student 
suggested  a  ceiling  mounted  monitor  so  students  could  follow  along  as  the  instructor  lectures. 
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Interviews:  Only  one  instructor,  teaching  in  the  Electronic  IG  class,  was  interviewed  during 
the  field  test  of  the  Paperless  Classroom  system.  The  instructor  was  very  confident  with  the 
material  and  enjoyed  teaching,  but  had  no  experience  lecturing  from  the  computer,  which  during 
this  study  was  his  main  source  of  information  for  the  lecture.  He  felt  hindered  from  walking  around 
the  classroom.  He  stated  he  preferred  to  walk  down  the  aisle  between  the  groups  of  tables.  He  felt 
he  could  have  closer  contact  with  the  students  and  make  sure  they  heard  his  lectures.  He  often 
carried  the  paper-based  TG  as  he  discussed  the  material.  The  instructor  felt  he  had  difficulties 
tracking  the  Electronic  IG  with  the  paper-based  TG. 

Test  Scores:  Test  scores  for  the  Electronic  IG  class  ranged  form  72-98%,  with  a  class  mean  = 
90.5.  Quiz  scores  were  not  available. 

Fully  Automated  Instruction 

Observation 

Learmng  Environment:  There  were  no  paper-based  instructional  materials  used  during  the 
class  or  given  to  students  for  study  purposes.  Students  used  the  computer  for  their  reading  and 
study.  Training  was  conducted  in  a  small  rectangular  room,  located  in  the  same  building  as  the 
classes  studied  during  Phase  I.  See  Figure  3  for  classroom  layout.  There  were  no  windows.  The 
room  was  separated  from  an  adjacent  classroom  by  an  accordion  folding  door,  mounted  between 
the  rooms.  The  room  was  well  insulated  with  grey  carpet  and  grey  fabric  covered  walls  and 
comfortably  air  conditioned.  The  fluorescent  lighting  was  adequate  and  pleasant.  A  pull  down  film 
screen  and  one  TV  monitor  (mounted  in  the  upper  right  comer)  were  in  the  front  of  the  classroom. 
A  VCR  and  one  closed-circuit  camera  were  placed  on  the  left  side  at  the  back  of  the  room.  One 
bookcase  was  at  the  front  left  of  the  room.  The  chairs  were  comfortable  with  padded  cushion  and 
chrome  arms. 

Student  workstations  consisted  of  an  embedded  and  angled  monitor,  covered  with  a  smoked 
colored  glass  shield  that  was  flush  with  the  table  top  (See  Figures  4  and  5).  This  provided  good 
visibility  for  students  and  instructor.  The  keyboard  tray  pulled  out  for  keyboard  adjustment.  A 
mouse  was  connected  to  the  side  of  the  keyboard.  The  workstations  were  grouped  in  pairs  on  each 
side  of  the  room  with  a  center  aisle  between. 

The  instructor  s  workstation  consisted  of  two  computer  monitors,  with  the  IG  displayed  on  one 
monitor  and  the  TG  displayed  on  the  other.  He  accessed  the  IG  and  TG  with  the  keyboard,  a  mouse 
or  trackball.  The  instructor  podium  was  at  the  front  of  the  classroom  with  a  white  board  mounted 
on  the  wall  behind.  The  instructor  placed  the  keyboard  and  mouse  on  the  podium.  His  workstation 
was  networked  with  student  workstations.  The  instructor  was  given  control  to  display  the  materials 
to  the  students  as  he  lectured.  Students  did  not  have  access  to  the  IG.  Graphics  were  displayed  on 
the  student  workstation  monitors.  In  traditional  classes,  the  instructor  illustrated  topics  with  poor 
quality  transparencies  and  schematic  drawings.  Students  had  difficulty  seeing  the  detailed 
transparencies  from  a  distance.  Now,  with  the  Paperless  Classroom  system,  graphics  were  more 
readable  on  the  computer  monitors.  The  instructor  directed  the  students  to  a  problem  sheet  or 
allowed  them  to  navigate  the  TG,  by  turning  “tracking”  on  or  off. 
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Figure  3.  Phase  II— Fully  automated  instruction  classroom  layout. 


Figure  4.  Phase  H— Wide  view  of  fully  automated  instruction 
student  workstation  design. 


Figure  5.  Phase  11— Fully  automated  instruction  student  workstation 
design. 


The  instructor  provided  system  training  for  the  students  during  the  course  indoctrination.  This 
was  not  Navy  standardized  instruction;  it  was  an  informal  training  session  to  familiarize  students 
with  the  workstation  and  software.  Students  were  excited  about  using  the  new  system  and  rapidly 
learned  how  to  scroll  through  the  text,  search  for  keywords,  size  and  zoom  graphics,  size  windows, 
use  the  TOC  and  on-line  calculator.  One  student  had  no  prior  computer  experience  and  quickly 
learned  how  to  use  the  various  features.  The  students  liked  the  new  workstations  and  the  capability 
to  view  the  graphics  from  their  monitor. 

A  few  problems  with  the  graphics  and  text  were  noted.  There  were  some  typos,  cases  of 
inconsistent  line  thickness  and  length,  different  type  fonts  being  used  for  text  within  a  graphic,  and 
some  graphics  needlessly  large.  Appendix  E  provides  examples  of  the  IG,  TG,  and  graphics. 

Classroom  Interaction:  The  instructor  was  an  enlisted  member  of  the  military.  He  was  very 
experienced  with  this  course  and  was  most  comfortable  using  the  system.  His  enthusiasm  for  the 
course  and  use  of  the  system  was  very  apparent.  The  instructor  interacted  well  with  students  and 
readily  responded  to  their  questions.  Students  felt  at  ease  with  each  other  and  the  instructor.  Most 
of  the  students  had  just  attended  their  first  level  of  training  at  “A”  school  and  did  not  have  fleet 
experience.  Several  students  knew  each  other  from  a  previous  course.  They  shared  experiences  and 
were  enthused  about  their  future  careers. 

Course  Activities:  The  evaluator  was  present  during  the  entire  course.  The  course  was 
conducted  during  normal  duty  hours  the  first  week.  During  the  second  week,  the  course  was 
conducted  from  noon  into  the  evening.  The  first  three  days  included  indoctrination  to  the  base  and 
the  training  center,  a  topic  about  safety,  and  time  to  take  care  of  personal  business.  Lecture  began 
on  the  fourth  day. 

The  same  topics  were  covered  as  those  taught  in  the  Fundamentals  course  observed  during 
Phase  I  of  this  study.  The  testing  procedures  were  the  same  as  those  used  during  the  earlier  classes 
observed  during  Phase  I. 

Survey 

Instructors  were  asked  to  respond  to  a  system  evaluation  survey  and  a  personalization  survey. 
Survey  results  are  presented  below. 

Instructor:  To  summarize  results  of  the  system  evaluation  survey  (Appendix  C,  Table  C-4), 
the  instructor  had  previous  use  with  computers  but  had  not  used  them  in  a  classroom  setting.  He 
had  five  years  experience  with  computers  and  had  been  teaching  for  several  years.  The  instructor 
supported  the  use  of  new  media  in  the  classroom  and  did  not  think  the  flow  of  information  was 
adversely  affected  during  his  lecture.  He  also  felt  confident  using  the  computer  and  agreed  there 
were  few  problems  with  the  hardware  and  software.  While  he  supported  the  use  of  electronic  aids 
for  personalizing  his  lecture  and  for  delivering  classroom  training,  he  did  not  think  the  course  could 
be  taught  in  less  time.  The  instructor  felt  strongly  that  the  use  of  this  system  will  be  useful  for  life 
cycle  management  of  the  course. 

There  was  a  section  for  additional  comments  at  the  end  of  the  survey.  The  instructor  suggested 
the  capability  to  display  graphics  to  a  wall  or  large  monitor  be  included  with  the  system.  He  wanted 
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to  see  a  better/easier  way  to  make  a  “bookmark”  in  the  on-line  materials.  A  bookmark  is  a  place 
holder  noted  by  an  icon  displayed  next  to  associated  text  on  the  computer  screen.  This  is  useful 
when  wishing  to  return  to  a  specific  topic  or  location  in  the  course  at  a  later  time.  He  suggested 
organizing  the  IG  topics  into  separate  electronic  books  so  it  would  be  easier  to  judge  the  time 
required  to  complete  a  topic  during  lecture.  The  instructor  really  enjoyed  being  able  to  interact 
better  with  the  students.  Being  able  to  insert  notes  or  personalization  into  the  TGs  for  students  was 
mentioned  as  something  very  useful  for  review  and  summary.  He  could  quickly  display  graphics 
as  he  reviewed.  The  instructor  thought  the  equipment  worked  fine,  but  suggested  an  additional 
monitor  be  placed  at  the  back  of  the  room.  He  liked  to  walk  among  the  students  as  he  lectured.  He 
would  then  be  able  to  glance  at  another  monitor  as  he  roamed  the  classroom. 

What  he  liked  least  was  that  students  tended  to  look  down  into  the  embedded  monitor  as  he  was 
lecturing.  Because  of  that,  he  found  it  difficult  to  maintain  eye  contact  with  them.  The  problem 
could  be  solved  by  an  overhead  display.  Another  comment  was  that  sometimes  the  system  was  a 
little  slow  to  display  a  graphic.  The  instructor  set  a  time  for  the  student  to  read  the  material  before 
class  because  they  were  not  given  printed  materials  to  take  away  for  study.  It  was  his  opinion  that 
students  got  lazy  about  reading  the  material  because  they  thought  the  computer  (keyword  search 
function)  would  find  the  answer  for  them  on  test  day.  Overall,  he  thought  the  Paperless  Classroom 
system  was  a  great  product  and  said  it  “could  be  fantastic  with  overhead  display.” 

Instructor  (Person^ization  Survey):  Seven  instructors  were  surveyed  about  the  time  they 
spent  personalizing  the  Fundamentals  IG.  These  instructors  were  qualified  to  teach  the 
Fundamentals  course,  but  only  two  were  teaching  at  the  time  of  our  study.  Some  of  the  questions 
were  not  answered  by  aU  seven  instructors  because  they  were  not  applicable.  It  was  important, 
however,  to  gather  information  from  all  of  them  concerning  the  use  of  the  Paperless  Classroom 
system  as  a  tool  for  entering  personalization.  The  survey  instrument  is  included  in  Appendix  D. 
Instructors  were  asked  to  estimate  the  time  spent  personiizing  an  IG  in  two  formats;  paper-based 
and  electronic.  Within  these  two  formats,  instructors  responded  to  questions  about  time  spent 
personalizing  and  later  updating  the  IG.  Not  all  of  the  instructors  had  experience  with  personalizing 
or  updating  both  the  paper-based  and  electronic  IG. 

As  reported  by  instructors,  time  spent  to  personalize  the  paper-based  or  the  electronic 
Fundamentals  IG  varied  greatly.  Since  the  electronic  IG  was  completed  in  May  1994,  instructors 
had  not  been  required  to  update  their  personalization.  Therefore,  there  were  no  data  to  report  about 
“updating”  the  Electronic  IG. 

We  were  also  interested  in  what  other  computer  systems  the  instructors  had  used  prior  to  using 
the  Paperless  Classroom  system.  Four  instructors  had  experience  with  IBM/PC  computers,  one 
instructor  had  experience  with  Commodores,  and  one  had  experience  with  Apple/Mac  computers. 
Five  instructors  responded  when  asked  to  designate  a  category  (beginner,  novice,  intermediate, 
accomplished,  advanced)  that  best  described  their  experience  with  the  Paperless  Classroom 
system.  The  experience  levels  were  varied  among  the  instructors;  Beginner  =  1;  Novice  =  1; 
Intermediate  =  2;  Accomplished  =  1. 

Student:  Nine  students  responded  to  the  system  evaluation  survey.  Eight  of  the  students  had 
previous  experience  using  computers  and  five  had  previously  used  a  computer  in  a  classroom 
setting.  When  asked  how  long  they  had  used  computers,  the  range  was  from  0-12  years.  The  survey 
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used  during  Phase  I  was  changed  to  find  out  what  computer  systems  students  had  used  in  the  past. 
The  majority  of  students  had  over  a  year  of  computer  experience  and  seven  of  nine  were  familiar 
with  IBM/PC  computers  and  five  reported  experience  with  PC  “Windows”  environment. 
Responses  made  concerning  system  functionality  were  favorable  (see  Appendix  C,  Table  C-5). 
Students  were  very  supportive  of  using  new  media  in  the  classroom  and  they  thought  the  use  of  a 
computer  made  the  course  more  interesting.  They  did  not  think  that  using  computers  in  the  class 
were  detrimental  to  their  learning  or  the  flow  of  information.  Four  students  were  “undecided”  about 
problems  with  the  software  and  time  lost  during  lecture  as  a  result  of  system  problems.  When  asked 
how  they  navigated  through  the  TG,  all  students  responded.  Four  students  reported  using  the  TOC, 
two  used  both  the  Search  feature  and  TOC,  and  three  used  all  available  methods. 

To  gain  an  overall  opinion  about  the  learning  environment,  students  were  given  various 
categories  and  asked  “What  do  you  like  MOST  about  the  course?”  Their  responses  are 
summarized  below. 

Classroom  Interaction:  Comments  from  six  students  revealed  they  thought  computers  greatly 
improved  classroom  interaction  and  made  the  course  more  interesting  for  everyone.  Students 
thought  there  was  more  class  participation  because  of  the  system.  They  praised  the  instructor's 
genuine  interest  in  the  course  material  and  his  capability  to  teach  from  the  system  and  readily 
respond  to  their  questions. 

Instructional  Strategies:  Six  students  commented  about  this  category.  Two  students  liked  the 
tracking  capability  the  instructor  had  so  he  could  control  where  they  were  in  the  TG  and  also  send 
graphics  to  their  monitors.  Another  two  students  thought  the  use  of  windows  for  graphic  display 
was  helpful.  Two  students  felt  they  could  navigate  through  the  text  very  quickly  and  thought  the 
course  material  was  easy  to  follow  and  understand. 

Equipment  Used:  AU  seven  students  who  commented  about  the  classroom  equipment 
considered  the  computers  “great,”  very  easy  to  use,  quick  for  retrieval  of  information.  They 
mentioned  that  the  workstations  were  particularly  nice  because  of  the  embedded  monitor  and  no¬ 
clutter  appearance  with  an  unobstructed  view  of  the  classroom.  Two  students  commented  they 
liked  the  system  capability  for  displaying  multiple  graphics  and  overlaying  on  an  instruction  sheet; 
this  made  it  easier  for  comparisons  of  tables  or  figures  and  promoted  better  understanding  of 
concepts. 

Subject  Matter/Curriculum:  Three  students  commented  they  found  it  easier  to  find  things 
using  the  system  search  feature  when  compared  with  “paper  tabs”  in  printed  course  materials.  They 
thought  the  course  was  well  organized  and  flowed  nicely. 

Instructor:  To  quote  a  student,  “He  [instructor]  enjoyed  the  computers  as  much  as  we  did,  so 
we  didn't  get  a  boring  monotone  instructor.”  Six  students  commented  they  thought  the  instructor 
was  very  informative,  had  a  great  sense  of  humor  (even  when  problems  occurred),  kept  the  class 
at  ease  and  assisted  with  all  questions  about  equipment  or  material.  Only  one  thought  he  needed 
more  training. 

Facilities:  Five  students  stated  that  the  facilities  were  “clean  and  very  sharp  looking,  neat, 
small  enough  to  have  a  personable  environment,  yet  comfortable.” 


20 


Students  were  Hsked  “What  do  you  like  LEAST  about  the  course?”  with  respect  to  the 
following  categories.  Their  comments  have  been  summarized  below. 

Classroom  Interaction:  Three  students  made  comments  about  this  category.  One  student 
commented  he  thought  the  class  was  held  up  by  someone  misusing  the  computer  and  getting 
sidetracked.  While  using  the  Paperless  Classroom  system,  the  instructor  had  control  of  what  was 
displayed  on  student  computers.  One  student  commented  that  if  a  student  was  taking  notes  about 
something  displayed,  the  instructor  might  choose  to  move  to  another  section  in  the  TG  before  the 
student  had  completed  his  note  taking.  This  is  a  control  issue  that  can  be  resolved  by  the  instructor 
asking  if  students  are  finished  reading  or  taking  notes  before  moving  on  to  another  topic. 

Instructional  Strategies:  Four  students  made  comments  about  this  category.  One  remarked 
that  “Sometimes  jumping  around  made  it  difficult  to  absorb  material.  We  should  be  able  to  hold 
onto  a  window  if  reading  it.”  One  student  thought  that  without  overhead  transparency  use,  it  was 
a  bit  harder  for  him  to  follow  along.  He  said,  “Instructor  did  his  best  by  drawing  diagrams  and 
following  through  that  way,  but  took  a  little  more  time  in  doing  so.”  Another  student  said,  “A  lot 
of  text  is  skipped  with  no  way  to  make  it  up  unless  you  come  in  after  hours.”  One  stated,  “Trying 
to  break  the  habit  of  having  a  book  to  take  home  and  review.  No  serious  problems.”  One  student 
suggested  that  more  real  life  examples  be  added  to  the  discussion. 

E)quipinent  Used:  One  student  said  he  couldn't  enter  his  notes  on  the  computer.  (At  this  time, 
there  is  no  capability  for  entering  text  on  the  student's  computer.)  One  student  stated  that,  “Not 
being  able  to  take  diagrams  out  of  text  while  in  the  search  path  makes  it  slower  to  search  through 
the  reading.  This  is  related  to  graphics  that  are  part  of  the  TG.  There  is  a  delay  in  the  program 
while  the  diagrams  are  drawn.  One  students  felt  there  needed  to  be  more  room  to  maneuver  with 
the  mouse;  the  mouse  cord  got  caught  in  keyboard  holder  when  stored.  One  commented  about  heat 
flow  problems;  too  much  heat  generated  (from  the  computer)  toward  the  user. 

Subject  Matter/Curriculum:  One  student  remarked,  “A  lot  of  material,  little  time.”  Another 
one  said,  “The  subject  matter  was  way  too  in-depth  for  a  1-1/2  week  course.”  Even  though  this  does 
not  seem  to  have  direct  relevance  to  an  evaluation  of  the  system  in  use,  the  fact  that  the  subject 
matter  is  very  in-depth  may  be  of  use  for  future  course  revisions.  A  major  revision  of  the  course 
would  impact  the  contents  and  structure  of  the  present  “electronic  book”  presented  by  the  Paperless 
Classroom  system. 

Instructor:  One  student  commented,  “just  needs  more  training.” 

Facilities:  No  comments. 

When  asked  “Do  you  have  suggestions  for  improving  the  use  of  this  technology  in  the 
classroom?”  all  nine  students  replied.  One  student  commented  he  would  like  to  see  the  system 
used  more  often,  prefers  it  over  books,  and  thought  the  advantages  outweighed  the  disadvantages. 
One  student  felt  strongly  that  an  electronic  overhead  display  should  be  added  to  the  Paperless 
Classroom  system.  This  would  allow  images  displayed  on  the  computer  to  be  projected  to  a  screen 
or  white  board  in  the  front  of  the  room,  much  as  instructors  have  done  with  transparencies 
displayed  with  an  overhead  projector.  This  would  be  particularly  useful  when  an  instructor  traces 
a  schematic  drawing  by  hand  during  a  discussion.  The  overhead  display  would  provide  the 
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schematic  to  all  students  at  once  with  an  increase  in  visibility.  Another  student  suggested  that  actual 
pictures  from  equipment  onboard  ship  be  included  with  the  graphics.  Six  students  commented 
about  enhancements  they  would  like  to  see:  a  personal  on-screen  note  pad  or  highlighting  markers 
for  text,  stationary  mouse  pad,  a  printer  so  notes  could  be  taken  home  for  study,  color,  and  a 
bookmark  function.  Another  stated  he  would  like  more  time  to  learn  about  system  features. 

Interviews:  One  instructor,  who  was  teaching  the  Fully  Automated  class  during  the  field  test 
of  the  Paperless  Classroom  system,  was  interviewed.  The  instructor  was  very  confident  with  the 
material  and  enjoyed  teaching  from  the  electronic  IG/TG.  He  liked  the  tracking  capability  the 
system  provided.  This  allowed  him  to  track  where  the  students  were  in  their  TG,  and  as  he  lectured 
he  could  display  associated  graphics.  The  instructor  quickly  found  it  beneficial  to  display  multiple 
graphics  overlaid  with  a  problem  sheet.  He  stated  learning  was  accomplished  in  much  less  time. 
There  were  a  few  minor  software  problems  but  no  lengthy  delays  occurred. 

Test  scores:  A  review  of  the  test  scores  for  the  Fully  Automated  class  (N  =  9)  revealed  a  range 
of  80-100%,  with  a  class  mean  =  91.67.  Quiz  scores  were  all  reported  to  be  100%.  The  cumulative 
course  mean  =  92.50.  This  mean  included  quiz  scores.  Test  scores  of  the  Fully  Automated  class 
were  compared  to  those  in  the  Electronic  IG  class.  The  results  of  an  analysis  of  variance  (ANOVA) 
found  no  significant  differences  between  the  Fully  Automated  and  Electronic  IG  classes,  F(l,29) 
=  .1558,  p  >  .05.  Table  4  presents  the  test  scores  for  both  classes. 

Table  4 


Phase  Il-Student  Test  Scores 


Class  Average  (%) 

Cum. 

Type  of  Instruction 

Quiz 

Exam  i 

Course 

Fully  Automated  (,N=  9) 

100 

92 

93 

Electronic  (N  =  22) 

81 

90 

89 

Course  critiques:  Course  critiques  were  not  available  during  Phase  n  of  this  study. 


Discussion  and  Conclusions 


Current  training  methods  are  rapidly  changing  due  to  advances  in  technology.  With  the 
expanded  availability  of  computers,  communications  networks,  and  peripheral  equipment, 
electronic  instruction  will  become  more  prevalent  in  classrooms  everywhere.  As  we  develop 
systems  and  evaluate  the  use  of  this  technology  in  the  classroom,  we  should  use  an  holistic 
approach.  By  this,  we  mean  that  in  addition  to  solving  an  identified  training  problem,  consideration 
must  be  given  to  the  needs  of  instructors  and  students,  the  ergonomic  design  of  the  electronic 
classroom,  the  social  environment,  overall  impact  on  the  educational  and  information  processes, 
and  impact  on  the  organization  (Bell  &  Elmquist,  1992;  Bruce,  1989;  Eisner  &  Carter,  1989; 
Roweton,  Wess,  &  Motley,  1988). 
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We  have  identified  a  need  for  more  efficiently  delivering  technical  information  in  the 
classroom.  Paperless  Classroom  addressed  this  problem  by  building  an  interface  to  this 
information  and  providing  the  instructor  a  tool  to  facilitate  delivery  of  electronic  training  materials. 
One  result  that  instructors  had  to  adapt  to  a  new  delivery  method  in  a  short  amount  of  time. 

In  a  study  on  instructors  using  multimedia,  Sammons  (1994)  found  that  faculty  members  were 
somewhat  hesitant  to  integrate  new  technology  into  the  instructional  process.  He  suggested  making 
equipment  readily  available  so  the  instructors  can  become  familiar  with  it  and  have  a  chance  to 
develop  their  materials  in  a  convenient  location.  A  survey  of  college  instructors  revealed  that  they 
were  reluctant  to  use  multimedia  in  the  classroom  because  of  a  lack  of  equipment,  lack  of  time,  and 
lack  of  knowledge.  These  same  factors  had  an  impact  on  use  of  the  Paperless  Classroom  system. 
The  instructors  we  interviewed  were  faced  with  time  constraints,  little  experience  with  the  system, 
and  lack  of  technical  support  In  spite  of  these  constraints,  they  supported  our  research  and  gave 
our  system  designers  excellent  feedback. 

Results  of  the  interviews,  observations,  and  opinion  surveys  revealed  that  the  prototype  system 
was  well  received  and  was  not  detrimental  to  the  learning  environment.  It  was  found  that  the 
students  and  instructors  surveyed  were  very  computer  literate.  Students  in  the  Fully  Automated 
class  had  no  problems  learning  to  use  the  system,  search  for  key  words,  scale  graphics,  and  size 
windows.  Within  a  short  amount  of  time,  students  were  very  comfortable  with  the  new  media  in 
the  classroom.  Instructors  were  generally  enthusiastic  about  using  new  media,  but  were  a  little 
reluctant  to  change  from  a  paper-based  to  an  automated  course.  With  their  already  busy  teaching 
schedules  and  other  military  duties,  this  new  system  required  additional  effort  on  their  part  to 
prepare  their  lessons  and  to  lecture  from  the  computer. 

Instructors  found  the  system  to  be  beneficial  as  they  prepared  to  lecture  by  personalizing  the 
IG,  but  some  were  not  convinced  this  was  a  time-saver.  The  variability  in  the  time  spent  completing 
personalization  could  be  attributed  to  the  instructor's  teaching  experience,  knowledge  of  the 
course,  computer  expertise,  availability  of  computers,  and  whether  the  notes  were  original  or 
borrowed  from  another  instructor.  Because  of  these  factors,  we  can  only  conclude  that  instructors 
will  become  more  comfortable  with  the  system  and  learn  ways  to  enhance  their  lectures,  realizing 
the  system’s  full  potential.  Electronic  personalization  will  be  saved  between  course  revisions  since 
the  notes  are  linked  to  the  discussion  points  by  the  software.  TTiere  is  no  need  to  copy  everything 
over  again  as  with  printed  materials. 

Test  scores  for  two  paper- based  classes  conducted  during  September  1993  were  obtained  to 
compare  with  the  other  two  groups  under  study.  Since  the  course  was  restructured  in  January  1994, 
test  scores  for  two  Fully  Automated  classes  and  seven  Electronic  IG  classes  have  been  compared. 
The  course  was  not  rewritten,  just  sequenced  better  with  the  text,  IG,  TG,  all  transparencies,  tables, 
and  figures  in  electronic  form.  A  comparison  of  these  test  scores  revealed  that  for  those  surveyed, 
students  attending  the  Fully  Automated  course  performed  as  well  as  those  students  attending  either 
the  Electronic  IG  or  the  paper-based  courses.  However,  the  number  of  students  attending  the  Fully 
Automated  Course  was  very  low.  We  had  expected  the  Fully  Automated  class  to  score  higher  on 
the  final  exam  than  the  other  two  class  formats.  The  fact  that  scores  were  only  comparable  might 
be  attributed  to  the  fact  that  students  in  the  Fully  Automated  class  did  not  have  paper-based  study 
materials  for  use  after  hours.  Also,  students  were  not  provided  with  the  capabilty  to  print  any  of 
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the  training  materials.  The  Paperless  Classroom  system  was  available  for  an  hour  before  class  each 
day,  but  few  students  came  to  study. 

Even  though  this  evaluation  is  limited  --  it  was  tested  on  only  one  course,  the  Paperless 
Classroom  system  was  successful.  The  system  was  a  useful  tool  for  instructors,  was  reliable,  and 
was  easy  to  use.  There  was  no  negative  impact  on  learning.  With  the  information  we  have  now,  we 
can  say  we  did  not  make  training  worse  by  automating  the  curriculum. 

As  the  use  of  electronic  aids  for  classroom  training  increases,  we  expect  more  efficient  use  of 
resources.  The  use  of  notebook  computers  for  after  hour  study,  links  to  laboratory  exercises  and 
remote  training  sites,  and  structured  training  for  instructors  and  students  on  the  use  of  these 
resources  will  become  prevalent.  It  is  anticipated  that  personnel  requirements,  technical  change 
management,  and  curricula  distribution  costs  will  be  reduced.  Electronic  delivery  will  lead  to 
reductions  in  publishing  costs  since  substantial  amounts  of  training  materials  will  no  longer  have 
to  be  printed. 


Recommendations 


It  is  recommended  that  AEGIS  Training  Center  automate  additional  courses;  as  more  technical 
manuals  become  available  in  electronic  form.  Instructors  and  students  should  be  given  more  on¬ 
site  training  on  the  use  of  Paperless  Classroom  system.  Navy  Instructor  Training  courses  should 
include  delivery  of  electronic  curriculum.  Consideration  should  be  given  to  linking  the  Paperless 
Classroom  system  to  laboratory  systems. 
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Appendix  A 
Phase  II 

Hardware  Configuration 


A-O 


Hardware  Confiauration 


A-l 


Appendix  B 


Phase  I 

Survey  Results 
Electronic  Instruction 


Table  B-1 

Phase  I  Instructor  Survey  Results 
Electronic  Instruction 


A.  BACKGROUND  INFORMATION 

1.  Rating:  FC  Rank:  E-5 

2.  Previous  computer  use?  Yes 

3.  Previous  use  of  computer  in  a  classroom?  no  response 

4.  How  long  have  you  used  computers?  4  yrs 

5.  Years  of  teaching  experience?  3  yrs 

B.  FUNCTIONALITY  EVALUATION 

Responses  made  by  instructor  based  on  the  following  scale. 

SA  A  U  D  SD 

strongly  agree  undecided  disagree  strongly 

agree  disagree 


1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information  in  the  course. 

2.  Use  of  the  computer  made  the  course 
more  interesting. 

3.  I  like  to  see  new  media  used  in  the  classroom. 

4.  I  felt  confident  when  using  the  computer. 

5.  There  were  few  problems  with  the  hardware. 

6.  There  were  few  problems  with  the  software. 

7.  Little  class  time  was  lost  because  of 
program  failures. 

8.  The  computer  should  be  used  in  other 
classrooms  to  enhance  training. 

9.  Use  of  the  computer  enhanced  student  learning. 

10.  I  could  personalize  the  Instructor  Guide  (IG) 

10.  I  could  personalize  the  Instructor  Guide  (IG) 
with  ease. 

11.  Development  time  to  prepare  the  IG  can 
be  reduced  by  using  this  system. 

12.  Training  on  using  the  computer  was  adequate. 

13.  Training  on  using  the  software  was  adequate. 

14.  I  will  be  able  to  cover  the  course 
material  in  less  time  by  using  this  system. 

15.  The  use  of  this  system  will  be  useful 

for  the  life  cycle  management  of  the  course. 

16.  Use  of  this  system  reduced  the  amount  of  time 
needed  for  personalizing  training  materials. 

17.  I  prefer  the  use  of  an  electronic  IG  when 
compared  to  a  paper  IG. 

18.  The  User  Manual  was  adequate. 


SA  A  U  D  SD 
X 


X 


X 

X 

X 

X 

X 

X 


X 

X 

X 

X 

X 

X 


X 


X 

X 


X 


B-1 


Table  B-2 

Phase  I  Student  Survey  Results 
Electronic  Instruction  (N=19) 


A.  BACKGROUND  INFORMATION 

1.  Previous  computer  use?  Yes=13  No=6 

2.  Previous  use  of  computer  in  a  classroom?  Yes=3  No=16 

3.  How  long  have  you  used  computers?  (yrs) 

0  =7 
1-4  =8 
6-12=4 

B.  FUNCTIONALITY  EVALUATION 

Number  of  student  responses  based  on  the  following  scale. 

SA  A  U  D  SD 

strongly  agree  undecided  disagree  strongly 

agree  disagree 


1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information  in  the  course. 

2.  Use  of  the  computer  made  the  course 
more  interesting. 

3.  I  like  to  see  new  media  used  in  the 
classroom. 

4.  The  instructor  seemed  confident  when  using 
the  computer. 

5.  There  were  few  problems  with  the  hardware. 

6.  There  were  few  problems  with  the  software. 

7.  Little  class  time  was  lost  because  of 
program  failures. 

8.  The  computer  should  be  used  in  other 
classrooms  to  enhance  training. 

9.  Teaching  from  the  electronic  IG 

did  not  adversely  affect  my  learning  the 
course  material. 


SA  A  U*  D  SD 

15  2  2 

2  5  9  2 


9  8  1 

5  10  4 

3  7  6  1 

2  8  8 

4  11  3 

5  9  4 

4  12  2  1 


*Some  felt  "undecided"  because  they  could  not  view  the  Electronic  Instructor  Guide. 
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Appendix  C 
Phase  II 

Classes  Conducted  and  Survey  Results 
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Table  C-1 


Phase  II 

AEGIS  Fundamentals  Classes  Conducted 
August  1993-July  1994 


Class  No. 

Conv-Grad  Dates 

Type  Instruction 

Students 

Final  Exam  Average 

93170 

23  Aug-7  Sep  93 

Paper-based 

10 

91% 

93180 

7  Sep-21  Sep  93 

Paper-based 

17 

90% 

94065 

18  Jan  94-1  Feb  94 

Electronic  IG 

23 

94% 

94090 

22  Feb  94-8  Mar  94 

Electronic  IG 

12 

94% 

94100 

22  Feb  94-8  Mar  94 

Electronic  IG 

20 

94% 

94130 

19  Mar  94-2  Apr  94 

Electronic  IG 

20 

93% 

94150 

11  Apr  94-22  Apr  94 

Electronic  IG 

22 

93% 

94180 

23  May  94-3  Jun  94 

Electronic  IG 

22 

91% 

94190 

31  May  94-14  Jun  94 

Fully  Automated 

9 

92% 

94200 

13  Jun  94-24  Jul  94 

Fully  Automated 

10 

82% 

94210 

27  Jun  94-12  Jul  94 

Electronic  IG 

24 

88% 

C-1 


Table  C-3 

Phase  n  Student  Survey  Results 
Electronic  IG  (N=22) 


A.  BACKGROUND  INFORMATION 


1.  Previous  computer  use?  Yes=17  No=5 

2.  Previous  use  of  computer  in  a  classroom?  Yes=8  No=14 

3.  How  long  have  you  used  computers?  (yrs) 

0  =5 
1-4  =7 
6-12  =  10 


B.  FUNCTIONALITY  EVALUATION 


Number  of  student  responses  based  on  the  following  scale. 

SA  A  U  D  SD 

strongly  agree  undecided  disagree  strongly 

agree  disagree 


1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information  in  the  course. 

2.  Use  of  the  computer  made  the  course 
more  interesting. 

3.  I  like  to  see  new  media  used  in  the 
classroom. 

4.  The  instructor  seemed  confident  when  using 
the  computer. 

5.  There  were  few  problems  with  the  hardware. 

6.  There  were  few  problems  with  the  software. 

7.  Little  class  time  was  lost  because  of 
program  failures. 

8.  The  computer  should  be  used  in  other 
classrooms  to  enhance  training. 

9.  Teaching  from  the  electronic  IG 

did  not  adversely  affect  my  learning  the 
course  material. 


SA  A  U* 

5  12  3 

3  8  9 

10  11 

7  13  1 

7  11  3 

8  10  3 

12  7  1 

8  10  3 

6  13  3 


*Some  felt  "undecided”  because  they  could  not  view  the  Electronic  Instructor  Guide. 


D  SD 
2 
1 


1 
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Table  C-4 

Phase  n,  Instructor  Survey  Results 
Fully  Automated  (N=l) 


A.  BACKGROUND  INFORMATION 

1.  Rating:  FCC  Rank:  E-7 

2.  Previous  computer  use?  Yes 

3.  Previous  use  of  computer  in  a  classroom?  No 

4.  How  long  have  you  used  computers?  5  yrs 

5.  Years  of  teaching  experience?  2-5  yrs 

6.  What  computer  systems  have  you  used?  PC,  Windows 

B.  FUNCTIONALITY  EVALUATION 


Responses  made  by  instructor  based  on  the  following  scale. 


SA  A  U 

D 

SD 

strongly  agree  undecided 

disagree 

Strongly 

agree 

disagree 

SA 

A 

u 

D 

1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information  in  the  course. 

2.  Use  of  the  computer  made  the  course 

X 

X 

more  interesting. 

3.  I  like  to  see  new  media  used  in  the  classroom, 

4.  I  felt  confident  when  using  the  computer. 

X 

X 

5.  There  were  few  problems  with  the  hardware. 

X 

6-  There  were  few  problems  with  the  software. 

X 

7.  Little  class  time  was  lost  because  of 

X 

program  failures. 

8.  The  computer  should  be  used  in  other 
classrooms  to  enhance  training. 

9.  Use  of  the  computer  enhanced  student  learning. 

X 

X 

10.  I  could  personalize  the  Instructor  Guide  (IG) 

X 

with  ease. 

11,  Development  time  to  prepare  the  IG  can 
be  reduced  by  using  this  system. 

12.  Training  on  using  the  computer  was  adequate. 

X 

X 

13.  Training  on  using  the  software  was  adequate, 

14.  I  will  be  able  to  cover  the  course 

X 

X 

material  in  less  time  by  using  this  system. 

15.  The  use  of  this  system  will  be  useful 

for  the  life  cycle  management  of  the  course. 

16.  Use  of  this  system  reduced  the  amount  of  time 

X 

X 

needed  for  personalizing  training  materials. 

17.  I  prefer  the  use  of  an  electronic  IG  when 
compared  to  a  paper  IG. 

18.  The  User  Manual  was  adequate. 

X 

X 
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Table  C-5 

Phase  n  Student  Survey  Results 
Fully  Automated  (N=9) 

A.  BACKGROUND  INFORMATION 

1.  Previous  computer  use?  Yes=8  No=l 

2.  Previous  use  of  computer  in  a  classroom?  Yes=5  No=4 

3.  How  long  have  you  used  computers?  (yrs) 

0  =2 
14  =3 
6-12  =  4 

What  systems  have  you  used?  PC  =  7  Apple/Mac  =  4  Commodore  =  1  Windows  =  5  Vax  Unix  =  1 

FUNCTIONALITY  EVALUATION 

Number  of  student  responses  based  on  the  following  scale. 

SA  A  U  D  SD 

strongly  agree  undecided  disagree  strongly 

agree  disagree 

SA  A  U  D  SD 

4  4  1 

7  2 

8  1 

3  5  1 

1  6  2 

4  4  1 

2  4  3 

3  6 

1  6  2 
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1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information  in  the  course. 

2.  Use  of  the  computer  made  the  course 
more  interesting. 

3.  I  like  to  see  new  media  used  in  the 
classroom. 

4.  The  instructor  seemed  confident  when  using 
the  computer. 

5.  There  were  few  problems  with  the  hardware. 

6.  There  were  few  problems  with  the  software. 

7.  Little  class  time  was  lost  because  of 
program  failures. 

8.  The  computer  should  be  used  in  other 
classrooms  to  enhance  training. 

9.  Teaching  from  the  electronic  IG 

did  not  adversely  affect  my  learning  the 
course  material. 


Appendix  D 
Survey  Instruments 


D-O 


PAPERLESS  CLASSROOM  SURVEY 
(Instructor) 

A,  BACKGROUND  INFORMATION 

1.  Rating/Rank: _ 

2.  Previous  computer  use?  Yes _  No _ 

3.  Previous  use  of  computers  in  a  classroom?  Yes _  No 

4.  How  long  have  you  used  computers? _ 

5.  Years  of  teaching  experience? _ 

6.  What  computer  systems  have  you  used?  MAC _  Apple _  PC _  VaxAJnix _  Windows _  Other 

B.  FUNCTIONALITY  EVALUATION 

Circle  the  response  that  best  describes  how  you  feel  about  each  statement. 

SA  A  U  D  SD 

strongly  agree  undecided  disagree  strongly 

agree  disagree 


1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information. 

2.  Use  of  the  computer  made  the  course  more 
interesting. 

3.  I  like  to  see  new  media  used  in  the  classroom. 

4.  I  felt  confident  when  using  the  computer. 

5.  There  were  few  problems  with  the  hardware, 

6.  There  were  few  problems  with  the  software. 

7.  Little  class  time  was  lost  because  of  program 
failures. 

8.  The  computer  should  be  used  in  other  courses 
to  enhance  training. 

9.  Use  of  the  computer  enhanced  student  learning. 


SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 

SA 

A 

u 

D 

SD 
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10. 

I  could  personalize  the  Instructor  Guide  (IG) 
with  ease. 

SA 

A 

u 

D 

SD 

11. 

Development  time  to  prepare  the  IG  can 
be  reduced  by  using  this  system. 

SA 

A 

u 

D 

SD 

12. 

Training  on  using  the  computer  was  adequate. 

SA 

A 

u 

D 

SD 

13. 

Training  on  using  the  software  was  adequate. 

SA 

A 

u 

D 

SD 

14. 

I  will  be  able  to  cover  the  course 
material  in  less  time  by  using  this 
system. 

SA 

A 

u 

D 

SD 

15. 

The  use  of  this  system  will  be  useful 
for  the  life  cycle  management  of  the 

SA 

A 

u 

D 

SD 

course. 

16. 

Use  of  this  system  reduced  the  amount  of  time 
needed  for  personalizing  training  materials. 

SA 

A 

u 

D 

SD 

17. 

I  prefer  the  use  of  an  electronic  IG  when 
compared  to  a  paper  IG. 

SA 

A 

u 

D 

SD 

18. 

The  User  Manual  was  adequate. 

SA 

A 

u 

D 

SD 

19.  Do  you  have  suggestions  for  improving  the  use  of  this  technology  in 
the  classroom? 


In  reference  to  the  topics  below,  what  do  you  like  MOST  about  the  course 
and  why? 

20.  Classroom  Interaction 


-2- 
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21.  Instructional  Strategies 


22.  Equipment  Used 


23.  Subject  Matter/Curriculum 


24.  Instructor 


25.  Facilities 


In  reference  to  the  topics  below,  what  do  you  like  LEAST  about  the  course 
and  why? 

26.  Classroom  Interaction 


27.  Instructional  Strategies 


28.  Equipment  Used 


29.  Subject  Matter/Curriculum 
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30.  Instructor 
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PAPERLESS  CLASSROOM  SURVEY 
(Student) 


A.  BACKGROUND  INFORMATION 

1.  Rating/Rank: _ 

2.  Previous  computer  use?  Yes _  No 


3.  Previous  use  of  computers  in  a  classroom?  Yes  No 


4.  How  long  have  you  used  computers? _ 

5.  What  systems  have  you  used?  MAC _  PC _  Apple _ Vax/Unix _ Windows _  Other 

B.  FUNCTIONALITY  EVALUATION 


Circle  the  response  that  best  describes  how  you  feel  about  each  statement. 


SA  A  U 

strongly  agree  undecided 

agree 

D 

disagree 

SD 

strongly 

disagree 

1.  Use  of  the  computer  did  not  adversely 
affect  the  flow  of  information  in  the  course. 

SA 

A 

U 

D 

SD 

2.  Use  of  the  computer  made  the  course 
more  interesting. 

SA 

A 

u 

D 

SD 

3.  I  like  to  see  new  media  used  in  the 
classroom. 

SA 

A 

u 

D 

SD 

4.  The  instructor  seemed  confident  when  using 
the  computer. 

SA 

A 

u 

D 

SD 

5.  There  were  few  problems  with  the  hardware. 

SA 

A 

u 

D 

SD 

6.  There  were  few  problems  with  the  software. 

SA 

A 

u 

D 

SD 

7.  Little  class  time  was  lost  because  of 
program  failures. 

SA 

A 

U 

D 

SD 

8.  The  computer  should  be  used  in  other 
classrooms  to  enhance  training. 

SA 

A 

U 

D 

SD 

9.  Teaching  from  the  electronic  IG 

SA 

A 

U 

D 

SD 

did  not  adversely  affect  my  learning  the 
course  material. 
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10.  Do  you  have  suggestions  for  improving  the  use  of  this  technology  in  the  classroom? 


In  reference  to  the  topics  below,  what  do  you  like  MOST  about  the  course 
and  why? 

IL  Classroom  Interaction 


12.  Instructional  Strategies 


13.  Equipment  Used 


14.  Subject  Matter/Curriculum 


15.  Instructor 


16.  Facilities 


In  reference  to  the  topics  below,  what  do  you  like  LEAST  about  the  course 
and  why? 

17.  Classroom  Interaction 


18.  Instructional  Strategies 


19.  Equipment  Used 


20.  Subject  Matter/Curriculum 


21.  Instructor 
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Questionnaire  on  Time  to  Personalize  Instructor  Guide  (IG) 


Todays  Date: _  Command/School: 

Name/Rank: _ 

Course  Name/CIN: 


INSTRUCTIONS 

(1)  Only  answer  as  many  of  the  sections  below  as  apply  to  you  for  THIS  course.  Fully  complete  the  questions 
within  the  sections  that  are  applicable  to  you  (i.e.,  Personalizing  a  Paper-based  IG  in  Part  1;  Personalizing  an 
Electronic  IG  in  Part  2;  or  both  Parts  1  and  2  if  you  did  both  of  these). 

(2)  If  You  Updated  an  IG  More  Than  Once:  If  you  received  changes  to  an  IG  that  caused  you  to  update  your 
previous  personalization  and  you  did  this  more  than  once,  please  use  another  one  of  these  forms  or  xerox  that 
section  and  write  a  note  on  it  to  indicate  that  it  was  the  2nd,  3rd,  etc.  time  you  did  this  work. 

(3)  Accurately  Estimating  the  Number  of  Hours:  To  arrive  at  an  accurate  estimate  of  your  work  on  an  IG, 
please  take  a  moment  to  consider  such  things  as  how  many  hours  you  work  a  day,  what  portion  of  the  day  you 
worked  on  an  IG,  and  then  how  many  days  you  worked  on  the  IG.  Then  total  the  number  of  hours. 

(4)  Accurately  Estimating  the  Percentage  of  a  Document  that  Changed:  To  arrive  at  an  accurate  estimate, 
think  of  the  number  of  lines  or  pages  that  were  changed  in  a  document  and  then  divide  that  by  the  total  number 
of  lines  or  pages  in  the  document 

(5)  Comments:  Please  add  any  relevant  additional  information  or  explanations  of  your  answers  on  the  form, 
e.g.,  descriptions  of  the  kind  of  changes  found  in  an  updated  IG. 
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Part  1 :  Personalizing  a  Paper-based  IG 


A.  Paper-based  IG:  Initial  First-Time  Personalization 

Complete  this  section  for  the  circumstance  where  you  initially  personalized 
a  Paper-based  IG  (e.g.,  when  you  first  taught  the  course  using  paper  materials). 

1.  Approximate  Date  when  work  was  performed: _ 

2.  Number  of  pages  in  Paper-based  IG: _ 

3.  Number  of  hours  required  to  enter  your  personalization  notes  into 

the  Paper  IG: _ 

4.  Hours  to  add  subsequent  minor  changes  after  original  personalization: _ 

5.  5a.  D-Yes  Cl-No  :  Was  the  information  you  entered  in  the  Paper /G 

developed  by  you  from  scratch? 

5b.  D-Yes  D-No  :  Did  you  use  a  previous  instructor’s  notes  as  an  aid  to  develop 

you  own  personalization  notes  in  the  Paper-based  IG  ? 

5c.  Percentage  of  total  time  devoted  to  developing  your  own  personalization 
notes: _ %. 

5d.  Percentage  of  total  time  devoted  to  using  someone  elses  IG  notes: _ %. 

B.  Paper  to  Paper  IG  Update: 

Complete  this  section  for  the  circumstance  where  a  new  version  of  a  Paper-based  IG  was 
received  and  you  had  to  convert  or  transcribe  your  personalization  from  the  old  version 
of  the  IG  to  the  new  version. 

1.  Approximate  Date  when  work  was  performed: _ 

2.  Number  of  pages  in  Paper-based  IG: _ _ _ _ 

3.  Number  of  hours  required  to  transcribe  your  personalization  notes  from  a  previous 
paper-based  IG  into  a  new  paper-based  version  of  a  IG: 

4.  Estimate  the  percentage  of  the  total  document  which  was  changed  between  the  previous  and 

new  version  of  the  IG: _  %  (i.e.,  what  percentage  of 

the  document  changed  irrespective  of  how  much  personalization  the  change  provoked) 

5.  Hours  to  add  subsequent  minor  changes  (after  original  update): 


C.  Paper  IG  Users:  Background  Information 


1. 


What  is  the  average  Prep-time  you  spend  before  each  lecture  (i.e.,  preparing,  rehearsing,  or 
reviewing  the  IG  or  TG)? 

_  hours 


2.  When  you  personalized  your  IG,  what  percentage  of  that  time  was  involved  with  consulting 
the  Trainee  Guide  (TG): _ %  (put  NA  if  there  is  no  TG  for  course) 


Part  1 :  Paper  IG 
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Part  2  :  Personalizing  an  Electronic  IG 


A.  Electronic  IG:  Initial  First-Time  Personalization 

Complete  this  section  for  the  circumstance  where  you  initially  entered  your 
personalization  in  the  Electronic  IGfor  the  first  time. 

1.  Approximate  Date  when  work  was  performed: _ 

2.  Number  of  pages  in  IG  in  its  paper  form: _ 

3.  Number  of  hours  required  to  enter  your  personalization  notes  into 

the  electronic  IG: _ 

4.  Hours  to  add  subsequent  minor  changes  after  original  personalization: _ 

5.  5a.  Ill-Yes  Ill-No :  Was  the  information  you  entered  in  the  Electronic  IG 

developed  by  you  from  scratch? 

5b.  lU-Yes  III-No :  Did  you  use  a  previous  instructor’s  Paper-based  IG  personalization 
notes  as  an  aid  in  developing  your  Electronic  IG  personalization? 

5c.  D-Yes  D-No  :  Did  you  use,  consult,  or  copy  a  previous  instructor’s  Electronic  IG 
personalization  in  developing  your  Electronic  IG  personalization? 

5d.  Percentage  of  total  time  devoted  to  developing  your  own  personalization 
notes: _ %. 

5e.  Percentage  of  total  time  devoted  to  using  someone  elses  IG  notes: _ %. 

6.  At  the  time  you  performed  this  work,  what  was  your  level  of  experience  with  the  system: 

□-Beginner  D-Novice  lU-Intermediate  D-Accomplished  D-Advanced 

B.  Electronic  to  Electronic  IG  Update: 

Complete  this  section  for  the  circumstance  where  a  new  updated  version  of  the  IG 
was  received  and  you  had  to  repersonalize  the  IG  because  it  had  changed. 

1.  Approximate  Date  when  work  was  performed: _ 

2.  Number  of  pages  in  IG  in  its  paper  form: _ 

3.  Number  of  hours  required  to  change  your  electronic  IG  personalization  in  order  to  update  it 

to  accommodate  the  new  electronic  version  of  the  IG: _ 

4.  Estimate  the  percentage  of  the  total  document  which  was  changed  between  the  previous  and 

new  version  of  the  IG: _ %  (i.e.,  what  percentage  of 

the  document  changed  irrespective  of  how  much  personalization  the  change  provoked) 

5.  Hours  to  add  subsequent  minor  changes  (after  original  update): _ 

6.  At  the  time  you  performed  this  work,  what  was  your  level  of  experience  with  the  system: 

□-Beginner  D-Novice  D-Intermediate  [D-Accomplished  d-Advanced 


Part  2:  Electronic  IG 
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C.  Electronic  IG  Users:  Background  Information 

Complete  this  general  background  information  if  you  used  the  Electronic  IG. 

Amount  of  experience  with  the  Electronic  IG  system: 

1.  How  long  have  you  used  the  Electronic  IG  system?  _ days  _ months 

2.  How  frequently  do  you  use  this  system? 

□-daily  D-a  few  times  a  week  D-onceaweek  Cl-a  few  times  a  month  D-monthly  or  less 

3.  How  long  did  it  take  you  to  learn  to  use  the  system?  _ days  _ months 

4.  What  is  your  current  level  of  experience  with  the  system: 

□-Beginner  d-Novice  d-Intermediate  CD-Accomplished  CD-Advanced 

5.  How  long  did  it  take  you  to  reach  that  level  of  experience  ?  _ days  _ months 


Computer  and  typing  experience: 

1.  What  is  your  level  of  keyboarding-typing  skill? 

CD-Hunt-and-Peck  CD-Novice  CD-Intermediate  CD-Advanced 

2.  How  many  words-per-minute  do  you  type: 

3.  How  long  have  you  used  computers  prior  to  using  this  system?  _ months 

4.  What  other  computer  systems  have  you  used  in  addition  to  this  system? 

CD-Apple/Mac  CD-UNK  CD-IBM/PC  CD-Other  CD-None 


Prep-Time  &  TG  Use: 

1.  What  is  the  average  Prep-time 
reviewing  the  IG  or  TG)? 


you  spend  before  each  lecture 
_  hours 


(i.e.,  preparing,  rehearsing,  or 


2.  When  you  personalized  your  IG,  what  percentage  of  that  time  was  involved  with  consulting 
the  Trainee  Guide  (TG): _ %  (put  NA  if  there  is  no  TG  for  course) 


Part  2:  Electronic  IG 
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STUDENT  CRITIQUE  SHEET 

COURSE  -  INSTRUCTOR 


UNIT 


IMPORTANT 


USi:  \0.  2  PENCIL  OSl\ 


•  EXAMPLE:  ca:  = 


•  ERASE  COMPLETELY  TO  CHANGE 


1 

-0=  ^1:  ^2=  =3:  -4^  -5-  -6=  ^7-  c8-- 

1 

c0=  -1=  =2-  ^3:  -43  c5=  ze-  ^7=  =8=  39- 

i  CDP 

303  cl:  32-'  C3:  C4C  35=  363  --7-  38:  39c| 

I  CODE 

eA3  cB3  cC-  ^D::  cfz  cja 

eL-  cRe  zQZ  zRe  -53  CT3  CU3 

eW3  cx-  -y-  ~z-  1 

1 

c83  zgz 

CLASS 

CQJ  C1E  C23  C3-  C43  C53  cB^  9  D 

1  NO. 

zQj  C1E  -23  z3e  c43  c53  zQz 

1 

zQz  -Id  c23  C;33  z43  c53  e63 

zQd  Cl3  C2-  =3: 


:4:=  =5=  ^6=  c?^  c8: 


LiiJ 


Comments  from  the  below  categories  will  be  used  to  improve  course  effectiveness.  Using  the  rating  scale 
provided,  blacken  the  box  that  best  expresses  your  critique  of  the  unit  of  instruction  just  completed.  Specific 
comments  are  requested  for  items  marked  'O  '  or  “U”. 


NAME 

RATE/RANK 

RATING  SCALE 

O  =  Outstandina 

INSTRUCTOR  fS^ 

G  =  Good 

M  =  Marainal 

U  =  Unsatisfactory 
N  =  Not  applicable 

ENVIRONMENT 


1 .  Classroom  space 

--  0 

-G 

-  M- 

u 

cN 

2.  Classroom  lighting  and  ventilation 

^  0  - 

-:G- 

-  M 

U- 

N 

3.  Desk/table  and  chair 

-Oz 

--G- 

-  M  • 

U3 

TRAINING  MATERIAL 


4. 

Quality  of  pubtications/manuais/prints/charts 

30- 

-G- 

-M3 

lU- 

:  N 

5. 

Quality  of  transparencies/slides/films/video  tapes 

zQz 

3G- 

M-- 

-  U  - 

-N 

Quality  of  instruction  sheets  {jco/info/problem/etc} 

30- 

3G3 

-M-- 

U  = 

:.N 

EQUIPMENT 

7.  Condition  of  training  aid  equipment  -0-  -G-  -M-  -U-  -N- 

8.  Condition  of  laboratory  equipment/special  tools _ cp,  -q-  -M:-  -U--  :::N^ 


i  INSTRUCTION 

I 

I  9.  Stress  placed  on  usage  of  publications 
1 10.  Stress  placed  on  safety  precautions 
j  11 .  Adequacy  of  break  time 

1 12.  Logical  arrangement  of  subject  matter 

113.  Time  allotted  for  classroom  instruction 
j14.  Time  allotted  for  laboratory  instruction 

jl5.  Availability  of  individual  assistance/extra  study 
1 16.  Adequacy  of  assistance  provided _ 


303 

3G3 

-IVl3 

::U3 

3N3 

303 

3G3 

3M3 

EU3 

3N3 

303 

3G3 

3M3 

CU3 

CN3 

303 

3G3 

-M3 

-U3 

3N  = 

303 

3G3 

-M3 

3U3 

3N3 

303 

3G3 

^M3 

CU3 

3N3 

303 

3G3 

-M3 
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INFORMATION  SHEET  T0007- 1-4-1 
RANDOM  ACCESS  MEMORY  (RAM) 


A.  INTRODUCTION 


The  purpose  of  this  infonnation  sheet  is  to  provide  you  with  required  information  on  random  access 
memory. 


B.  REFERENCES 

1.  Mano,  M.  Morris,  Computer  System  Architecture 


C.  INFORMATION 

1.0  RANDOM-ACCESS  MEMORIES  (RAM) 

1.0.1  A  memory  unit  is  a  collection  of  storage  registers,  together  with  the  associated  circuits  needed 
to  transfer  information  in  and  out  of  the  registers.  Memory  registers  can  be  accessed  for  information 
transfer  as  required  and  hence  the  name  random-access  memory,  abbreviated  RAM. 

1.0.2  A  memory  unit  stores  binary  information  in  groups  of  bits  called  words.  Each  word  is  stored 
in  one  memory  register.  A  word  in  memory  is  an  entity  of  n  bits  that  move  in  and  out  of  the  memory 
unit.  A  word  of  eight  bits  is  sometimes  called  a  byte. 

1.0.3  A  memory  word  is  a  group  of  O’s  and  I’s  and  may  represent  a  number,  an  instruction  code, 
alphanumeric  characters,  or  any  other  binary-coded  information. 

1.0.4  The  communication  between  a  memory  unit  and  its  environment  is  achieved  through  control 
lines,  address  selection  lines,  and  data  input  and  output  lines.  Tie  control  signals  specify  the  direc¬ 
tion  of  transfer  required.  That  is,  whether  a  word  is  to  be  stored  in  a  memory  register  or  whether  a 
word  previously  stored  is  to  be  transferred  out  of  a  memory  register.  The  address  lines  specify  the 
particular  word  chosen  out  of  hundreds  or  thousands  available.  The  input  lines  provide  the  informa¬ 
tion  to  be  stored  in  memory  and  the  output  lines  supply  the  information  coming  out  of  memory.  A 
block  diagram  of  a  memory  unit  is  shown  in  Figure  1-1. 

Rgure  1-1  Block  diagram  of  a  memory  unit 

1.0.5  A  memory  unit  is  specified  by  the  number  of  words  it  contains  and  the  number  of  bits  in  each 
word.  The  address  selection  lines  select  one  particular  memory  word  out  of  the  m  words  available. 
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Each  word  in  a  memory  is  assigned  an  identification  number,  called  an  address,  starting  from  0  and 
continuing  with  1, 2, 3,  and  up  to  m-1.  The  selection  of  a  specific  word  inside  the  memory  is  done  by 
inserting  its  binary  address  value  into  the  selection  lines.  A  decoder  inside  the  memory  unit  accepts 
this  address  and  opens  the  paths  needed  to  select  the  word  specified.  Thus,  k  address  bits  can  select 
any  one  of  2k  =  m  words.  Computer  memories  may  range  from  1024  words,  requiring  an  address  of 
10  bits,  to  1,048,576  =  2^  words,  requiring  20  address  bits.  It  is  customary  to  refer  to  the  number  of 
words  in  a  memory  unit  with  the  unit  K.  K  refers  to  1024  =  2^°  words:  thus  IK  =  1024  words,  4K  = 
4096  words,  and  64K  =  2*®  words. 

1.0.6  The  two  control  signals  are  called  "read"  and  "write".  A  write  signal  specifies  a  transfer  in 
operation;  a  read  signal  specifies  a  transfer-out  operation.  On  accepting  one  of  the  control  signals, 
the  internal  control  circuits  inside  the  memory  unit  provide  the  desired  function.  When  the  memory 
imit  receives  a  write  control  signal,  the  internal  control  transfers  the  n  data  input  bits  into  the  word 
specified  by  the  address  lines.  With  a  read  control  signal,  the  word  selected  by  the  address  lines 
appears  in  the  n  data  output  lines. 

2.0  ICRAM 

2.0.1  Integrated  circuit  memories  sometimes  have  a  single  line  for  the  readAvrite  control.  One 
binary  state,  say  1,  specifies  a  read  operation  and  the  other  binary  state  specifies  a  write  operation.  In 
addition,  one  or  more  enable  lines  may  be  included  in  each  IC  package  to  provide  means  for  expand¬ 
ing  several  packages  into  a  memory  unit  with  a  larger  number  of  words. 

2.0.2  The  internal  construction  of  a  random-access  memory  of  m  words  with  n  bits  per  word  con¬ 
sists  of  m  X  n  binary  storage  cells  and  the  associated  logic  needed  to  select  a  word  for  writing  or 
reading.  The  binary  storage  ceU  is  the  basic  building  block  of  a  memory  unit.  The  logic  diagram  of  a 
binary  ceU  that  stores  one  bit  of  information  is  shown  in  Figure  2-1.  Although  the  ceU  is  shown  to 
include  five  gates  and  a  flip-flop,  internally  it  is  constructed  with  a  two-transistor  flip-flop  having 
multiple  inputs.  The  binary  ceU  of  a  memory  unit  must  be  very  smaU  in  order  to  be  able  to  pack  as 
many  ceUs  as  possible  in  the  semiconductor  area  avaUable  in  the  chip.  The  binary  ceU  is  shown  to 
have  three  input  lines  and  one  output  line.  The  purpose  of  the  select  input  is  to  select  one  ceU  out  of 
the  many  avaUable.  With  the  select  line  at  1,  a  1  in  the  read/write  terminal  forms  a  path  from  the  out¬ 
put  of  the  flip-flop  to  the  ouq)ut  terminal.  With  the  read  write  terminal  at  0,  the  bit  in  the  input  line  is 
transferred  into  the  flip-flop.  Both  the  input  and  output  arc  disabled  when  the  select  line  is  0.  Note 
that  the  flip-flop  operates  without  a  clock  pulse  and  that  its  purpose  is  to  store  the  information  bit  in 
the  binary  ceU. 


Rgure2-1  Memory  ceU. 

2.0.3  The  configuration  of  a  4  by  3  RAM  is  shown  in  Figure  2-2.  It  consists  of  four  words  of  three 
bits  each  for  a  total  of  12  binary  storage  ceUs.  Each  smaU  box  labeled  BC  in  the  diagram  includes 
within  it  the  circuit  of  a  binary  ceU.  The  four  lines  included  with  each  BC  box  designate  the  three 
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(a)  Logic  diagram 


(b)  Block  diagram 


Figure  2-1  Memory  cell. 
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inputs  and  one  output  as  specified  in  the  diagram  of  Figure  2-1. 

2.0.4  The  two  address  lines  go  through  a  2  by  4  decoder  with  an  enable  input  When  the  memory 
enable  is  0,  all  the  outputs  of  the  decoder  are  0  and  none  of  the  memory  words  are  selected.  Wifli  the 
memory  enable  input  at  1,  one  of  the  four  words  is  selected,  depending  on  the  bit  combination  of  the 
two  address  lines.  Now,  with  the  rcad/write  control  at  1,  the  bits  of  tl^  selected  word  go  through  the 
three  OR  gates  to  the  output  terminals.  The  non-selected  binary  cells  produce  O’s  in  the  inputs  of  the 
OR  gate  and  have  no  effect  on  the  outputs.  With  the  read/write  control  at  0,  the  information  available 
in  the  input  lines  is  transferred  into  the  flip-flops  of  the  selected  word.  The  non-selected  binary  cells 
in  the  other  words  are  disabled  by  their  selection  line  so  their  i^evious  values  remain  unchanged. 
Thus,  with  the  memory  enable  at  1,  the  rcad/write  control  initiates  the  required  read  and  write  opera¬ 
tions  for  the  memory  unit  An  inhibit  operation  is  obtained  by  maintaining  the  memory  enable  at  0. 
This  condition  leaves  the  contents  of  all  words  in  memory  as  they  were,  irrespective  of  the  value  of 
the  read  write  control. 

Figure  2-2  4  by  3  IC  RAM  (BC  is  the  binary  cell  of  figure  2-1). 

2.0.5  IC  RAMs  sometimes  employ  binary  cells  whose  outputs  can  be  tied  to  form  a  wire-OR  or  a 
wire-AND  function.  Other  IC  RAMs  provide  tri-state  outputs.  These  outputs  are  convenient  when 
two  or  more  ICs  are  connected  to  form  a  memory  unit  with  a  larger  number  of  words  since  they 
eliminate  the  need  for  external  OR  gates  that  would  otherwise  be  needed. 
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Figure  2-2  4  X  3  IC  RAM  (BC  is  the  binary  cell  of  figure  2-1). 
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